The Transformer: attention and generation

The formula on the slide is the whole mechanism. Each token is projected into a query, a key and a value. One token’s query is scored against the key of every visible token, the scores are scaled by the square root of the key dimension and passed through a softmax, and the resulting weights mix the values. Several heads run in parallel so different heads can track different relations (Vaswani et al., 2017). Self-attention means Q, K and V all come from the same sequence.

Where it came from

Sequence-to-sequence models squeezed the whole source sentence into one fixed vector with a recurrent network (Sutskever et al., 2014). Attention let the decoder look back at every source position instead (Bahdanau et al., 2015). The Transformer kept that as cross-attention and dropped the recurrence, so training runs over all positions at once and parallelises on GPUs.

Two shapes

Encoder-decoder. The encoder reads the source, the decoder writes the target through cross-attention. Translation and summarisation models were built this way.

Decoder-only. Prompt and answer share one sequence, and a causal mask hides later tokens from earlier ones. Most current chat and code models are decoder-only. Position is no longer added as sinusoids as in the 2017 paper; many models use rotary position embeddings (RoPE), which rotate the queries and keys by position.

The costs that reach production

Attention compute grows with the square of the sequence length, though the other layers dominate until contexts get long. The KV cache stores each earlier token’s keys and values so a decode step runs only the new token through the layers, at the price of GPU memory that scales with context. Both facts turn into the latency and cost numbers in part 7.

Generation is append-only

The loop on the right of the slide: embed the prompt, run the decoder blocks, take the next-token probabilities, sample one token, append it, repeat. Nothing already emitted is revised. Two consequences for the system you own. Cap output length, because the loop stops only when the model emits a stop token or you cut it. And put reasoning before the verdict in any schema you ask for.

Where it goes wrong

A review schema with verdict before reasoning. Without a separate thinking phase the verdict is written first, and the reasoning that follows is written to fit it.

What to do

• Order output fields so the evidence comes before the conclusion.
• Set a maximum output length on every call and check the stop reason.
• Treat context length as a cost, not a free resource. Part 6 covers what it does to quality.