Tokens, embeddings and vector space

The slide has two pipelines. The top one runs inside the language model: text, tokenizer, token IDs, token embeddings, Transformer layers. The bottom one runs beside it: a passage or a query, an embedding model, one vector, a cosine search over a vector index. They share a word, embedding, and almost nothing else.

Tokens

A tokenizer splits text into subword units using byte pair encoding or a similar scheme (Sennrich et al., 2016). Each model family has its own vocabulary, so the same text counts differently across providers. English runs at roughly three to four characters per token. Code and numbers usually take more tokens per word.

The gap across languages is larger. Petrov et al. (2023) found translations of the same text differing by up to 15 times in token count. For anything serving Tamil or Hindi beside English, that is a cost and a context-window difference, not a rounding error. Inside the model, each token ID selects one row of a learned embedding matrix, and that is what the Transformer layers read.

Embeddings

A separate embedding model compresses a passage into one vector of hundreds to a few thousand dimensions. Retrieval ranks passages by cosine similarity, which is the dot product for unit-length vectors. Nearness reflects whatever that model learned to group, which is mostly topic. “The endpoint requires auth” and “the endpoint does not require auth” can be nearest neighbours.

Approximate nearest-neighbour indexes trade recall for speed. Measure recall@k on your own queries instead of trusting the index defaults, and size passages at a few hundred tokens rather than whole documents.

Where it goes wrong

A team switches embedding models for new documents only. If the dimensions match, nothing errors. The scores between old and new vectors are meaningless, and old documents rank as noise from that day on.

What to do

• Count tokens with the provider’s tokenizer, never with string length. Limits, bills and latency all run on tokens.
• Change the embedding model and re-embed everything, unless the provider documents a shared space.
• Record the model name and version on every stored vector, so a partial re-embed is detectable.
• Tune passage size on recall@k, not on what looks tidy.