What to measure for your use case
Accuracy is only one number. Task success, faithfulness, safety, speed, cost and user satisfaction all matter. Decide what "good" means before you build.
In 60 seconds
What to measure for your use case
Accuracy is only one number. Task success, faithfulness, safety, speed, cost and user satisfaction all matter. Decide what "good" means before you build.
The main dimensions
| Dimension | Question it answers | Example measure |
|---|---|---|
| Task success | Did the user get what they came for? | Share of conversations where the ticket was resolved without a human |
| Correctness | Is the answer right? | Share of answers matching a checked reference |
| Faithfulness | Does the answer stick to the given sources? | Share of claims supported by the retrieved documents |
| Safety | Does it avoid harmful or disallowed output, without refusing good requests? | Rate of unsafe outputs on a red-team set, and rate of wrong refusals |
| Latency | Is it fast enough? | Time to first word and total time, at the median and the slowest 5% of requests |
| Cost | Can we afford it at scale? | Average cost per conversation or per resolved task |
| User satisfaction | Do people like it and come back? | Thumbs up rate, repeat use, complaints |
From a vague goal to a measurable one
- 1
Start with the job
"Help farmers understand crop insurance claims in Hindi and Marathi." - 2
Name what failure looks like
Wrong deadline, wrong document list, answer in the wrong language, rude tone, invented scheme names. - 3
Pick measures
Correct deadline and document list; answer in the user's language; no invented facts; answer in a few seconds. - 4
Set targets
Decide a minimum for each, and which ones are must-pass (for example, no invented scheme names). - 5
Agree and write it down
Product, engineering and domain experts sign off before building.
Must-pass versus nice-to-have
- "The bot should give good answers."
- One overall score that averages everything
- Only measured by the team that built it
- "At least a set share of answers have the correct deadline and document list, checked against the official scheme page."
- Separate scores per dimension, with must-pass items listed
- Checked by domain experts and by real users
Metrics that can be gamed
Watch and read more
Lab
A one-page evaluation plan that defines "good" for one AI feature.
The problem
Pick a feature, for example "a WhatsApp bot that answers questions about a city's property tax in English, Hindi and Kannada".
- Write the job in one sentence and list five ways it could fail.
- For each of the seven dimensions (task success, correctness, faithfulness, safety, latency, cost, user satisfaction), write one measure, or "not needed, because …".
- Mark each measure as must-pass or trade-off, and give a rough target.
- For your top two measures, write a counter-metric.
- List which groups you will break results down by (for example, language).
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1A new model raises correctness on your test set but doubles cost and latency. How do you decide?Reveal
Go back to your written targets. If the old model already met the correctness target and the new one breaks the cost or latency budget, stay. If the old model missed a must-pass correctness target, the new one may be worth it, perhaps only for the hard questions (routing easy questions to a cheaper model). The decision should follow the criteria you agreed, not the biggest number.
Q2Why is user satisfaction alone a poor measure of an AI assistant's quality?Reveal
Users often cannot tell whether an answer is correct, especially in areas like law, tax or health. They may rate confident, friendly, long answers highly even when they are wrong. Satisfaction is important, but it must be paired with correctness and safety checks by people who can judge the content.
Questions people ask
Do I really need all seven dimensions?
Not always, but go through each one and decide. An internal tool may not need a satisfaction survey. A public health bot needs strong safety and correctness measures. Writing "not needed, because …" is still a decision.
What targets should I set if I have no data yet?
Start with what the current process achieves, for example how accurate human agents are today, or what users accept now. Your first targets can be rough. Revise them once you have real measurements, but write down why you changed them.
Who should decide what "good" means?
Not only engineers. Include the product owner, domain experts (doctors, lawyers, agronomists, support leads) and, if possible, real users. They know which mistakes are harmless and which are serious.
How do latency and cost fit with quality?
They are part of quality for the user and the business. A perfect answer that takes a minute may be useless on a phone with a weak connection. A very accurate model may cost too much per conversation. Measure them next to accuracy, so trade-offs are visible.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.