Why retrieval beats retraining
A model only knows what it saw in training. Retrieval lets it read your fresh and private documents at answer time, and show where each answer came from.
In 60 seconds
Why retrieval beats retraining
A model only knows what it saw in training. Retrieval lets it read your fresh and private documents at answer time, and show where each answer came from.
The basic RAG loop
- 1
Prepare (once, then keep updated)
Split your documents into chunks, and store them in a search index with their source and date. - 2
Search
When a question comes in, find the few chunks most likely to contain the answer. - 3
Build the prompt
Put the question and the chunks together, each chunk with an id such as [S1], [S2]. - 4
Generate
The model writes an answer using only those chunks, and cites them. - 5
Show sources
The user sees the answer and can click through to the original text.
Why this usually wins
| Need | Retrieval (RAG) | Retraining or fine-tuning |
|---|---|---|
| Fresh information | Update the index today; the next question sees it | Train again, test again, deploy again |
| Private documents | Stay in your own store; you control who can search what | Knowledge is baked into the weights for anyone who can use the model |
| Citations | Each answer can point to the exact chunk it used | The model cannot reliably say where a fact came from |
| Removing a fact | Delete the document and its chunks | Very hard to make a trained model truly forget one fact |
| Cost to start | An index and a search step | Training data, compute and evaluation for each change |
When fine-tuning is the right tool
- Facts that change: prices, rules, stock, schedules
- Knowledge where you must show the source
- Documents only some users are allowed to see
- Anything you may need to delete later
- A fixed style, tone or output format
- A narrow, repeated task such as classifying tickets
- Making a small model do one job well and cheaply
- Domain wording the base model handles badly
Watch and read more
Lab
A one-page decision note: RAG, fine-tuning, both or neither, for three real-looking use cases.
The problem
For each use case below, write 4–6 lines: which approach you would choose (RAG, fine-tuning, both, or neither), and why. Mention freshness, sources, privacy and cost.
- A hospital's staff bot that answers questions from internal duty rosters and procedure notes. (Do not design medical advice for patients.)
- A tool that turns rough notes from field officers into a fixed, three-part inspection report format.
- A support bot for a phone company that answers questions about current recharge plans.
Finally, write one use case of your own where plain prompting, with no retrieval and no fine-tuning, is enough.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1A manager says: "Let's fine-tune on all our PDFs so the model just knows everything." Give two strong reasons against this as the only plan.Reveal
First, the knowledge will go stale: every change in the PDFs needs new training and testing, while an index can be updated the same day. Second, the model cannot cite where an answer came from, and cannot respect per-user permissions: anything trained in is available to everyone who uses the model, and hard to delete later. Fine-tuning may still help with tone or format, alongside retrieval.
Q2When might "put everything in the prompt" be a reasonable first version?Reveal
When the document set is small and stable, for example one short policy of a few pages, and everyone who uses the bot may read all of it. Then a long prompt is simple and gives a useful baseline. Keep a small test set so you can compare later, and switch to retrieval when the set grows, changes often or has permissions.
Questions people ask
Does RAG stop hallucinations completely?
No. It reduces them when the right text is found, because the model has the facts in front of it. But the search can miss, and the model can still misread or add things. That is why this track covers citations, "not found" answers and measurement.
Is RAG only for chatbots?
No. The same retrieval step powers search pages, "related articles", support tools that suggest answers to human agents, and agents that look things up before acting.
Do I need a special vector database to start?
Not always. For a few thousand chunks, a simple in-memory search, or a search extension in a database you already run, can be enough. Choose a dedicated tool when size, speed or features really need it, and check its current docs.
Can I put all my documents in the prompt instead?
For a small, fixed set of pages, sometimes yes, and it is a fair baseline to compare against. As the collection grows, cost and speed get worse, and the model may miss details. Retrieval keeps the prompt short and focused.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.