Latency and cost: TTFT and TPOT
Latency has two budgets, driven by different parts of the request. Cost is price times volume, and resent input usually dominates an agent's bill.
The pipeline on the slide is a request’s life: network and queue, prefill of the input, reasoning if the model does any, the first token, then one decode step per output token. Two numbers describe it, and they have different fixes.
Two numbers
TTFT, time to first token, covers the network, the queue, a parallel prefill of the whole prompt, and any hidden reasoning tokens a reasoning model decodes before its first visible token. It grows with input length.
TPOT, time per output token, is the mean gap between tokens during decode. Each step is memory-bound, grows with model size, context length and server batch size, and can emit more than one token where speculative decoding is used.
Latency is roughly TTFT plus TPOT times the visible output tokens minus one.
Two illustrative cases. At 25 milliseconds TPOT and a 0.5 second TTFT, a 400 token reply takes about 10.5 seconds, nearly all of it decode. A 5 token label from a 40,000 token prompt with a 3 second TTFT finishes in about 3.1 seconds, nearly all of it prefill. Same model, opposite fixes.
Cost
Cost is roughly input tokens times the input price plus output tokens times the output price, before cache rates. Output tokens, reasoning included, are often priced at four to eight times input. That puts a chat reply’s cost in its output.
An agent loop is different. A ten step loop that starts at 5,000 tokens and adds 2,000 per step resends 140,000 input tokens to reach a 23,000 token context, because every step sends the whole context again. Prompt caching discounts that growth. It does not remove it.
The levers
The table on the slide is a price list, and every row has a second column.
- Streaming cuts the perceived wait. It costs you late validation, since you cannot check a reply you have already shown.
- Compact output cuts decode time and cost. It costs detail, and a hard cap can truncate.
- Prompt caching cuts TTFT and input cost. It costs a fixed prompt order, because only an exact prefix hits. A timestamp at the top of the system prompt defeats it.
- A smaller model cuts TPOT and cost. It costs quality, so check the evals.
- A batch API cuts the price by about half. It costs up to 24 hours of waiting.
What to do
- Short answers: cut TTFT. Long answers: cut TPOT. Agent loops: cut resent input.
- Set a p95 latency target per feature and measure it per feature, not per endpoint.
- Check current prices, including cache write and read rates, before you design the loop.
Short answers: cut TTFT. Long answers: cut TPOT. Agent loops: cut resent input. Set p95 targets per feature.