Evals and observability
Evals are the test suite for the probabilistic parts. Traces are the debugger.
The pipeline on the slide is a loop, not a line. Traces show failing cases. Failing cases become the eval set. A fix passes it. CI reruns the evals and blocks the release on a regression. A canary deploy catches what the suite missed, and its traces start the loop again.
Offline evals
Build cases from real traffic, edge cases and incidents. Prefer code graders: a schema check, an exact field match, a numeric tolerance. Where you need an LLM judge, check it against human labels first, and grade retrieval separately from the answer so a bad answer can be traced to a bad passage. For agents, measure task success, step count, tool errors and cost per task, not only the final text.
Run the suite on every change to a prompt, model, tool or index. One green run of a sampled model proves little. Repeat each case, five is a reasonable number, and gate on drops beyond the run-to-run noise against the last release.
Online evals and traces
Canary deployments, A/B tests, user feedback and drift alerts cover what the offline set has not seen yet.
Open one trace per request, a child span per step, and a grandchild per model call, tool call or guardrail decision. Spans carry the model and prompt versions, tokens, cost, stop reason, tool arguments and errors, with personal data redacted. Propagate the trace context into tools and workers, or the multi-agent run from part 11 becomes unreadable. OpenTelemetry’s semantic conventions for generative AI name the spans invoke_agent, chat and execute_tool; they are still marked Development, so pin the version you adopt. Open-source options include promptfoo, Ragas, DeepEval and Langfuse.
Where it goes wrong
A model judging its own outputs. Self-preference bias can inflate the scores (Panickssery et al., 2024).
What to do next week
- Write 20 eval cases from real traffic.
- Schema-check one output.
- Propagate trace IDs into one tool.
- Gate one consequential action on human approval.
That is the whole series in four lines. Part 3 showed how a Transformer generates text. Parts 4 to 7 priced tokens in latency and cost. Parts 8 to 11 let the model act inside limits that code enforces. Parts 12 to 14 chose prompting, retrieval or fine-tuning, and added guardrails, evals and traces.
Every production failure becomes an eval case that blocks its return.
- Panickssery, Bowman and Feng, 2024. LLM Evaluators Recognize and Favor Their Own Generations.
- OpenTelemetry. Semantic conventions for generative AI systems.