Guardrails and fallbacks
Treat model output like untrusted user input.
Three boxes on the slide are drawn in sienna: the input guard, the output check and the action gate. The model sits between them in ink, and the fallback ladder runs underneath. The rule that organises all of it: the model is a component you do not trust, in a system you do.
Three gates
Input. Authentication, rate limits, scope checks, personal data redaction, injection screening.
Output. Schema and business-rule checks, which are deterministic and can block on their own. Grounding checks against retrieved passages and policy classifiers, which are probabilistic and need thresholds. Stop reason and truncation checks, from part 5.
Action. Allowlists and least privilege, spend caps, sandboxes, and human approval for anything consequential.
Injection
Indirect prompt injection (Greshake et al., 2023) hides instructions in the documents and tool results a model reads. OWASP’s Top 10 for LLM Applications keeps prompt injection at LLM01, and its Agentic Top 10 lists agent goal hijack at ASI01. Screening lowers the risk and never removes it. The privileges must hold in code that a hijacked model cannot change: if the model can only call tools on an allowlist, with the user’s own permissions, an injected instruction has nowhere to go.
The fallback ladder
On failure, climb one rung at a time. Retry transient errors with backoff and jitter, and retry writes only with an idempotency key. Repair once by returning the exact validation error to the model. Then a fallback model, held to the same checks. Then rules, a template or a cached answer. Then a person.
The ladder is different for answers and actions. Answers may degrade: a cached order status is acceptable. Actions fail closed: a cached refund never is. No valid output, no write. Circuit breakers and per-step timeouts bound the damage when a provider slows down.
Where it goes wrong
A fallback model whose prompt was tuned for the primary model and never evaluated. The first outage floods the escalation queue.
What to do
- Enforce privilege in code, not in the prompt.
- Log every guardrail decision with a reason code. Part 14 turns those logs into evals.
- Evaluate the fallback path as seriously as the primary one.
Treat model output and tool results as untrusted: actions fail closed, answers may degrade.
- Greshake et al., 2023. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.
- OWASP GenAI Security Project. OWASP Top 10 for LLM Applications.