Evaluating AI systemsMiddleLesson 2176 min read

Benchmarks and their limits

What popular benchmarks such as MMLU, HumanEval, SWE-bench and LMArena actually measure, why they wear out, and why a high score does not mean the model can do your job.

✦ Ask the tutor🧪 Lab · 30 min📝 Lesson test
Lesson in motion

In 60 seconds

Benchmarks and their limits

What popular benchmarks such as MMLU, HumanEval, SWE-bench and LMArena actually measure, why they wear out, and why a high score does not mean the model can do your job.

1/5
In simple words
A student who tops the maths olympiad may still struggle to fill in a railway reservation form. The olympiad measures one thing very well. It does not measure everything. AI benchmarks are the same: each one is a test of one kind of skill.
A benchmark is a fixed set of test questions with a fixed way of scoring answers. Everyone runs the same test, so scores can be compared across models. That is useful. But a benchmark answers a narrow question: how well does this model do on these items, scored this way? It does not answer will it work for my users?

Four well-known benchmarks, in plain words

BenchmarkWhat the model doesHow it is scoredWhat it does not tell you
MMLUAnswers multiple-choice questions across many school, university and professional subjects.Share of questions answered correctly.Whether it can explain, write, or handle open questions. Picking A–D is not the same as doing the work.
HumanEvalWrites short Python functions from a description.The code is run against unit tests. Often reported as pass@k: the chance that at least one of k tries passes.Whether it can work in a large, messy codebase, or write code that is secure and easy to maintain.
SWE-benchFixes real issues taken from open-source Python projects on GitHub.The model's code change must make the project's tests pass. There are variants, such as a human-checked subset.Whether it can do your team's work, in your language, with your tools and review standards.
LMArena (formerly Chatbot Arena)People type a prompt, see two anonymous answers and vote for the one they prefer.Votes are turned into a ranking, similar to chess ratings.Whether answers are correct. Votes also reward style, length and tone, and reflect who is voting and what they ask.
Note
We give no scores here on purpose. Scores change with every new model, and the same benchmark can be run in different ways (number of examples in the prompt, exact wording, how answers are parsed). Two reported numbers are only comparable if the setup was the same.

Saturation: when a test stops being useful

When a benchmark is new, models score badly and differences are large. Over time, scores climb. When most leading models score near the top, the benchmark is saturated: it can no longer tell them apart. A small gap near the top may be noise, or may come from a few questions that have wrong answer keys. Researchers have found labelling errors in some popular benchmarks, so a perfect score is not always even possible.
  1. 1

    New benchmark

    Hard for current models. Scores are spread out and informative.
  2. 2

    Scores climb

    From real progress, from tuning to the test, and sometimes from leaked test items.
  3. 3

    Saturation

    Top models are bunched near the ceiling. Differences are mostly noise.
  4. 4

    Replacement

    A harder benchmark appears, and the cycle starts again.

Benchmark versus your task

Reading a leaderboard badly
  • "It ranks first, so it is best for our Hindi customer-support bot."
  • Comparing numbers from different papers with different setups
  • Treating one number as general intelligence
Reading a leaderboard well
  • Using it to make a shortlist of models to test on your own data
  • Checking the exact setup: prompt, number of examples, scoring script
  • Asking which skill the benchmark tests, and whether you need that skill
Public benchmarks are mostly in English and test general skills. Your task might be summarising Marathi court orders, answering farmers' questions about crop insurance, or extracting fields from GST invoices. No public benchmark measures that exactly. The rest of this track shows you how to build the test that does.
Watch out
A leaderboard is also a target. When many teams tune their models to climb one leaderboard, the score can rise faster than the real skill. This is Goodhart's law: when a measure becomes a target, it stops being a good measure.
Try this. Pick an AI feature you use or build. Write down which public benchmark is closest to it, and three ways your real task is different from that benchmark.

Watch and read more

Datasets on Hugging Face (search for a benchmark by name)Hugging Face · reading

Lab

A one-page "benchmark fit" note for one real task you care about.

~30 min

The problem

Choose a task, for example "answer customer questions about bus ticket refunds in Hindi and English".

  1. Pick one public benchmark you think is closest. Find its dataset card or paper and read at least five real test items.
  2. Write down: what skill it tests, the answer format, the language, and how it is scored.
  3. Make a two-column table: What the benchmark tests and What my task needs. Add at least five rows (language, answer length, tools, domain knowledge, safety, tone).
  4. End with one sentence: "This benchmark can help me … but it cannot tell me …".

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1A new model scores a few points higher than your current model on MMLU. Your task is extracting fields from Tamil invoices. Should you switch?Reveal

Not on that evidence. MMLU tests multiple-choice knowledge in English, which says little about Tamil text, invoice layouts or field extraction. A few points near the top may also be noise. Run both models on your own invoice test set, and compare accuracy, cost and latency before deciding.

Q2Why can a benchmark that was excellent five years ago be misleading today, even if nothing about it changed?Reveal

Three reasons. Models may have saturated it, so it no longer separates them. Its items may have leaked into training data across the web, so scores reflect memory. And teams may have tuned models towards it, so scores rise faster than real skill. The test stayed the same, but its relationship to real ability changed.

Please sign in to continue.

Questions people ask

If benchmarks are so limited, why does everyone report them?

Because they are cheap, public and repeatable. They are good for a first shortlist and for tracking general progress. The mistake is using them as the final decision for a specific product.

Is LMArena better than fixed benchmarks because real people vote?

It measures something different: which answer people prefer in a quick side-by-side look. That is useful, but voters may not check facts, may prefer longer or nicer-looking answers, and may ask very different questions from your users. Treat it as one signal among several.

Where can I find a benchmark's actual questions?

Many benchmarks publish their data and scoring code, often on Hugging Face or GitHub, with a paper on arXiv. Read a few real items before you trust a score. You will often be surprised by what is actually being tested.

Should I build my own benchmark instead?

For a product, yes: a private test set built from your real task (Lesson 4). Use public benchmarks to narrow the list of models, then your own tests to decide.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress