Context windows and sampling

The pipeline on the slide is one decode step: the context window goes into the model, the model produces logits, the logits are divided by the temperature, a softmax turns them into probabilities, a top-p filter trims the tail, one token is sampled and appended, and the step repeats.

One budget

The context window holds everything: the system prompt, the tool definitions, the conversation history, any documents you attach, and the answer. A large tool catalogue shrinks the room for the answer.

When a reply hits the maximum output tokens, the call still succeeds. The stop reason says length or max_tokens, a JSON object or tool call ends mid-field, and it fails later in a parser far from the cause. Check the stop reason on every call.

The model is stateless. Even when the API stores the conversation for you, every earlier token counts toward the window and is billed as input again on each turn, discounted only where a cache hits. Part 7 prices this.

Sampling

The model produces a logit z for every token in its vocabulary. A temperature below 1 widens the gaps between logits so probability concentrates on the top token. Above 1 flattens the distribution. Near 0 is near greedy.

Top-p, also called nucleus sampling, keeps the smallest set of tokens whose cumulative probability reaches p (Holtzman et al., 2020), so the candidate set adapts to the model’s confidence, where top-k keeps a fixed count. Tune one of temperature or top-p, not both. Many current models reject non-default values with an error, and some degrade below the default, so read the model’s documentation before touching either.

Zero is not deterministic

Temperature 0 is not a determinism guarantee. The server’s batch size changes with load, the kernels are not batch-invariant, and a near tie between two tokens can flip between runs. Tests built on temperature 0 pass until they do not.

Where it goes wrong

A test suite that asserts exact output at temperature 0. Green for weeks, then flaky under production load with no code change.

What to do

• Check the stop reason and treat a truncated reply as a failure, not a short answer.
• Keep the default sampling settings unless the model docs advise otherwise for your task.
• Gate on pass rates over repeated runs, five is a reasonable start, not on one run.