Prompting, RAG or fine-tuning

Three levers, one decision tree. Prompting changes the instructions and examples. Retrieval-augmented generation, RAG, changes the context per query: embed document chunks, add the top matches to each prompt. Fine-tuning changes the model’s weights. The table on the slide compares what each is best for, what it needs, what it costs per call and how it fails.

Diagnose first

The decision tree starts at the same place every time: a prompt, an output schema and a small eval set. When the evals fail, ask why.

• Unclear task or wrong format: fix the prompt, and use a JSON schema or the provider’s structured output mode.
• Missing, private or stale facts: add retrieval.
• Behaviour that still misses after prompting, or cost at volume: fine-tune.

Retrieval done properly

Chunk documents along their headings. Keep each chunk’s source and its permissions, and enforce those permissions at retrieval time with the end user’s identity, not the service account’s. A small, stable corpus can sit in a cached prefix until it grows. Part 6 showed why long inputs cost accuracy as well as tokens.

Fine-tuning, and what not to use it for

Fine-tune for behaviour: a consistent format, a narrow task, a tone. Tuned model tokens often cost more, and when the base model is retired you retrain.

Do not fine-tune to teach facts. Fine-tuning on new knowledge was linked to more hallucination (Gekhman et al., 2024), and retrieval beat unsupervised fine-tuning on new facts, with fine-tuning plus retrieval trailing retrieval alone (Ovadia et al., 2024).

Where it goes wrong

An assistant fine-tuned on last quarter’s refund policy. Re-indexing one document would have fixed it.

What to do

• Prompt and eval first.
• Retrieve facts. Fine-tune for behaviour.
• Eval the combination, and grade retrieval separately from the answer.