Measuring retrieval and answer quality
A RAG system can fail at search or at answering. Measure both: recall@k and MRR for retrieval, faithfulness and correctness for answers, on a golden question set.
In 60 seconds
Measuring retrieval and answer quality
A RAG system can fail at search or at answering. Measure both: recall@k and MRR for retrieval, faithfulness and correctness for answers, on a golden question set.
The golden question set
| Field | Example |
|---|---|
| Question | "Can I carry forward unused casual leave?" |
| Relevant chunk ids | hr-leave-v3#1 |
| Reference answer | No. Unused casual leave cannot be carried to the next year. |
| Tags | policy, English, simple |
- Use real questions from support tickets, search logs or staff, with personal data removed.
- Cover your mix: languages and scripts, short and long questions, codes and names, follow-ups, and tricky ones.
- Include questions with no answer in the documents. The right output is "not found".
- Keep it versioned, and do not tune on it so much that it stops being a fair test. Many teams keep a separate held-out part.
Retrieval metrics
| Metric | Question it answers | How it is computed |
|---|---|---|
| Recall@k (hit rate) | Did the right chunk appear in the top k at all? | Share of questions where at least one relevant chunk is in the top k |
| Recall@k (full) | How many of all relevant chunks were found? | Relevant chunks in top k ÷ all relevant chunks, averaged over questions |
| MRR (mean reciprocal rank) | How high was the first right chunk? | Average of 1 ÷ (rank of first relevant chunk); 0 if none in the list |
| Precision@k | How much of the top k was useful? | Relevant chunks in top k ÷ k |
def hit_at_k(results, relevant, k):
return 1.0 if any(r in relevant for r in results[:k]) else 0.0
def recall_at_k(results, relevant, k):
return len(set(results[:k]) & set(relevant)) / len(relevant)
def reciprocal_rank(results, relevant):
for rank, r in enumerate(results, start=1):
if r in relevant:
return 1.0 / rank
return 0.0
# Made-up results from a search system, for three golden questions.
golden = [
{"q": "carry forward leave?", "relevant": ["hr#1"], "results": ["hr#0", "hr#1", "it#4"]},
{"q": "travel allowance per day", "relevant": ["tr#2", "tr#3"], "results": ["tr#2", "hr#0", "tr#9"]},
{"q": "wifi password reset", "relevant": ["it#7"], "results": ["it#4", "it#5", "hr#2"]},
]
k = 3
n = len(golden)
print("hit@3 ", sum(hit_at_k(g["results"], g["relevant"], k) for g in golden) / n)
print("recall@3", round(sum(recall_at_k(g["results"], g["relevant"], k) for g in golden) / n, 3))
print("MRR ", round(sum(reciprocal_rank(g["results"], g["relevant"]) for g in golden) / n, 3))Answer metrics
- Faithfulness (groundedness): is every claim supported by the retrieved sources? An answer can be faithful and still wrong, if the sources were wrong or old.
- Correctness: does the answer match the reference answer?
- Citation accuracy: does each cited source really support its sentence?
- Correct "not found": for questions with no answer, did the system say so instead of inventing one?
LLM-as-a-judge, with care
- 1
Write a clear rubric
Narrow yes-or-no questions work better than "rate from 1 to 10". - 2
Give the judge the evidence
The question, the sources and the answer, and the reference answer where you have one. - 3
Check the judge against people
Have people label a sample too, and measure agreement, for example with Cohen's kappa. - 4
Keep people in the loop
Review disagreements and a random sample of every run. - 5
Fix the judge setup
Keep the judge prompt and model fixed between runs, so scores stay comparable.
Watch and read more
Lab
A 20-question golden set and a script that scores a search system on hit@k, recall@k and MRR.
The problem
Use the metric code from the lesson (Python 3, standard library). Choose a small document set you may use, such as your own notes or a public FAQ page you chunk yourself, or reuse the documents from the hybrid search lab.
- Write 20 golden questions with relevant chunk ids and a one-line reference answer. Include at least 3 with no answer, 3 in an Indian language or romanised form, and 2 with codes or names.
- Run your search from an earlier lab (or rank by hand, honestly) and record the top 5 results for each question.
- Compute hit@5, recall@5 and MRR overall, and per tag.
- Pick the three worst questions. For each, say whether the failure is in extraction, chunking, search or the question itself.
- Write a 5-line rubric an LLM judge could use for faithfulness.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1How should no-answer questions be handled in recall@k and MRR, and why?Reveal
They have no relevant chunks, so recall and reciprocal rank are undefined for them (recall would divide by zero). Leave them out of the retrieval metrics and measure them separately at the answer level: did the system correctly say "not found"? Mixing them in, for example by counting them as zero, would unfairly lower retrieval scores and hide the real behaviour you care about.
Q2A new re-ranker raises MRR from 0.61 to 0.64 on your 40-question golden set. Should you ship it?Reveal
Not on that number alone. On 40 questions, a small change can come from a few questions moving, which may be noise. Look at which questions changed and why, check per-tag scores for any group that got worse, check latency and cost, and if possible test on more questions or a held-out set. Ship if the gain is consistent, no important group regresses, and the extra latency is acceptable.
Questions people ask
How many golden questions do I need?
Even 30 to 50 good, varied questions catch many problems and are far better than none. Grow the set over time from real failures. For comparing two close options reliably, you need more, and you should look at how much scores vary.
Should I measure retrieval or answers first?
Retrieval first. It is cheaper and more objective, and if the right chunk is not retrieved, the answer cannot be grounded in it. Then measure answers on top.
Can a model write my golden questions?
It can help draft them from your documents, but model-written questions often look too much like the document text and are easier than real ones. Mix in real user questions and have a person review every item.
Are there ready-made RAG evaluation tools?
Yes, several open-source libraries compute these metrics and run judge prompts. They can save time. Understand what each metric really measures, check it against human labels, and remember that tools and their defaults change.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.