Context limits and context drift
The window is a capacity limit, not a quality guarantee. Design what goes in instead of filling it.
A model that accepts 128,000 tokens does not use 128,000 tokens well. Two studies put numbers on the gap, and the design pattern on the right of the slide is the response.
Effective is smaller than advertised
Liu et al. (2023) moved the document holding the answer through a set of 20 retrieved documents and found a U-shaped curve: models used evidence at the start and the end of the context well and evidence in the middle badly. GPT-3.5-Turbo did worse with the answer in the middle than with no documents at all.
NoLiMa (Modarressi et al., 2025) removed literal word overlap between the question and the evidence, so the model had to understand rather than match. Of 13 models claiming 128K or more, 11 fell below half their short-context score by 32K tokens. The curves differ by model, so test your own at your real length.
Drift
Drift is what happens to a long conversation. Old turns and stale facts accumulate until they outweigh the instructions. A tool result pasted whole into history keeps costing tokens and attention on every later turn. Over the limit, hosted APIs usually reject the request, but framework buffers and local servers configured to a smaller window may cut the front of the prompt silently, and the front is where the instructions were.
The design pattern
The diagram shows a context builder that assembles each call from parts instead of appending forever.
- A token budget per section, with an output reserve held back.
- Stable content first: instructions, tools, then history. An identical prefix between turns keeps prompt caching hitting.
- State, evidence and constraints after that, re-injected from a state store rather than trusted to survive in history.
- Tool results trimmed to the fields that matter before they enter history.
- Old turns compacted in occasional batches rather than every turn, because each rewrite invalidates the cache.
Summaries are lossy. Exact values such as IDs, amounts and dates live in the state store, which part 11 makes the orchestrator’s single source of truth.
Where it goes wrong
A coding agent pastes a 20,000 token test log into history on turn 3. By turn 12 it edits the migrations folder it was told to leave alone, because the instruction is now a small fraction of what it can see.
What to do
- Budget every section of the prompt and log the totals per call.
- Test at your real context length and turn count, not at the demo length.
- Put exact values in state, never in summaries.
The window is capacity, not quality: budget every section and test at your real length and turn count.
- Liu et al., 2023. Lost in the Middle: How Language Models Use Long Contexts.
- Modarressi et al., 2025. NoLiMa: Long-Context Evaluation Beyond Literal Matching.