A/B tests and online metrics
Offline evals tell you what might happen. A/B tests with real users tell you what does happen. Learn guardrail metrics, sample size intuition and novelty effects.
In 60 seconds
A/B tests and online metrics
Offline evals tell you what might happen. A/B tests with real users tell you what does happen. Learn guardrail metrics, sample size intuition and novelty effects.
| Offline evals | Online (A/B) tests | |
|---|---|---|
| Data | Your fixed test set | Real users, real traffic |
| Speed | Minutes to hours | Days to weeks |
| Risk | No user sees a bad version | Some users see the new version |
| Measures | Correctness, format, safety on known cases | Behaviour: resolution, retention, complaints, cost |
| Main weakness | May not match real use | Slow and noisy; needs enough users |
How an A/B test works
- 1
Pass offline evals first
Never A/B test a version that fails your must-pass checks. - 2
Choose one main metric
Decide in advance, for example "share of chats resolved without a human". - 3
Choose guardrail metrics
Things that must not get worse: complaints, safety flags, latency, cost per chat. - 4
Split users randomly
Each user is assigned by chance to A (current) or B (new), and stays in that group. - 5
Run for a fixed time
Decide duration and sample size before starting. Do not stop early the moment B looks good. - 6
Decide
Ship B only if the main metric improves and no guardrail gets worse.
Guardrail metrics
Sample size intuition
def users_per_group(p, d):
"""Rule of thumb: about 80% power, 5% significance, two groups."""
return round(16 * p * (1 - p) / d ** 2)
print(users_per_group(0.20, 0.02)) # detect 20% -> 22%: 6400 per group
print(users_per_group(0.20, 0.05)) # detect 20% -> 25%: 1024 per groupTraps to avoid
| Trap | What happens | Defence |
|---|---|---|
| Peeking | Checking results every day and stopping the moment B looks better finds "wins" that are just noise. | Fix the duration and sample size in advance. |
| Novelty effect | Users try a new feature because it is new; usage drops later. | Run long enough; look at the trend over time, and at returning users. |
| Change aversion | Users first dislike any change, even a good one. | Same: run long enough to see behaviour settle. |
| Too many metrics | Look at 30 metrics and a few will "improve" by chance. | One main metric chosen in advance; treat the rest as clues. |
| Unequal groups | One group gets more Hindi users, or more weekend traffic. | Random assignment by user, and check the groups look similar. |
Watch and read more
Lab
A one-page A/B test plan with metrics, guardrails and a sample size estimate.
The problem
Scenario: a railway enquiry bot resolves about 25% of chats without a human. You have a new prompt that you hope raises this.
- Write the main metric exactly (what counts as "resolved"?).
- List three guardrail metrics and why each matters.
- Using the rule of thumb, compute the users per group needed to detect a rise to 27% and to 30%. Show your working.
- Given about 2,000 chats a day, estimate how many days each test would take. Is either practical?
- Write how long you will run the test to handle novelty effects, and your rule for stopping.
Practice lab
Edit the code and run it. Python runs in your browser — nothing is sent to our server.
No output. Add print(...) to see values.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your A/B test shows the new prompt raised resolution, but complaints also rose slightly, though not by a statistically clear amount. What do you do?Reveal
Do not ship on autopilot. A guardrail moving the wrong way is a warning even if it is not clearly significant. Read a sample of the complaints and the resolved chats in group B. If the new prompt is closing chats with wrong answers, the "win" is false. You may need a longer test, a better resolution definition, or a fix before trying again.
Q2Why might an offline eval improvement disappear in an A/B test?Reveal
The test set may not match real traffic: real users use different languages, shorter questions and different topics. The improvement may be in cases that are rare in practice. Users may react to things the test set does not measure, like speed, length or tone. And the offline gain may have been noise on a small test set. The A/B test measures the real mix.
Questions people ask
If offline evals are good, why run an A/B test at all?
Because your test set is a guess about real use. Real users ask different things, react to tone and speed, and change their behaviour. An A/B test measures the actual outcome. Many teams find that offline wins shrink or even reverse online.
Should I randomise by message or by user?
Usually by user, so each person has one consistent experience. Randomising each message lets one person see both versions, which confuses them and mixes up the measurement.
What if I do not have enough users for an A/B test?
Then rely more on offline evals, careful human review of real conversations, and staged rollouts where you watch guardrail metrics closely. Be honest that you cannot detect small effects.
Can I use thumbs up and thumbs down as my main metric?
It is a useful signal, but few users click, and those who do may not be typical. Pair it with behaviour-based metrics, such as whether the problem was solved or the user came back with the same issue.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.