Evaluating AI systemsMiddleLesson 2225 min read

Human evaluation done right

Human judgement is the gold standard only when it is done carefully: clear rubrics, more than one rater, a check on agreement, and blind comparisons.

✦ Ask the tutor🧪 Lab · 40 min📝 Lesson test
Lesson in motion

In 60 seconds

Human evaluation done right

Human judgement is the gold standard only when it is done carefully: clear rubrics, more than one rater, a check on agreement, and blind comparisons.

1/5
In simple words
In a cricket match, one umpire can make a mistake, so there are two on the field and a third watching replays. When people judge AI answers, we also use more than one judge and check whether they agree.
Humans can judge things rules and models struggle with: whether advice is actually useful, whether a translation sounds natural, whether a tone is respectful. But people are also inconsistent. They get tired, read instructions differently and have their own preferences. Good human evaluation is designed to reduce these problems.

Four habits of good human evaluation

  1. 1

    Clear rubric

    Specific questions with examples of each level, the same rubric for every rater.
  2. 2

    Right raters

    People who know the domain and the language; train them on examples first.
  3. 3

    More than one rater

    At least on a sample, so you can measure agreement.
  4. 4

    Blind comparisons

    Raters do not know which model or version produced which answer, and the order is shuffled.

Measuring agreement: Cohen's kappa

If two raters agree 80% of the time, is that good? It depends. If almost every answer is "good", they would agree often just by chance. Cohen's kappa corrects for this. It compares the observed agreement with the agreement you would expect by chance, given how often each rater uses each label.
Kappa = (observed agreement − chance agreement) ÷ (1 − chance agreement). A kappa of 1 means perfect agreement. 0 means no better than chance. Below 0 means worse than chance.
Python: Cohen's kappa for two raters (standard library only)python
from collections import Counter

def cohens_kappa(a, b):
    """Cohen's kappa for two raters who labelled the same items."""
    if len(a) != len(b) or not a:
        raise ValueError("need two non-empty lists of the same length")
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    count_a, count_b = Counter(a), Counter(b)
    labels = set(count_a) | set(count_b)
    chance = sum(count_a[k] * count_b[k] for k in labels) / (n * n)
    if chance == 1:
        return float("nan")  # both raters used one identical label: kappa is undefined
    return (observed - chance) / (1 - chance)

rater_1 = ["good", "good", "good", "good", "good", "good", "bad", "bad", "bad", "bad"]
rater_2 = ["good", "good", "good", "good", "good", "bad", "bad", "bad", "bad", "good"]
print(round(cohens_kappa(rater_1, rater_2), 2))  # 0.58
Work it through by hand. The raters agree on 8 of 10 items, so observed agreement is 0.8. Each rater said "good" 6 times and "bad" 4 times, so chance agreement is (6×6 + 4×4) ÷ 100 = 0.52. Kappa = (0.8 − 0.52) ÷ (1 − 0.52) ≈ 0.58. So 80% raw agreement is only moderate once chance is removed.
Note
People often use rough bands, for example treating values above about 0.6 as substantial agreement. These bands are conventions, not laws. What counts as "enough" depends on the task. Cohen's kappa is for two raters; for more raters, look up measures such as Fleiss' kappa or Krippendorff's alpha.

When raters disagree

Low agreement is information, not failure. Read the items where raters disagreed. Usually the rubric is unclear ("is a correct answer in the wrong language good or bad?"), or the item itself is ambiguous. Fix the rubric, add a worked example, and measure again. Once agreement is good, a single rater per item may be enough, with a second rater on a regular sample.
Weak human evaluation
  • One person rates everything, knowing which answer is from the new model
  • "Rate overall quality 1 to 5" with no examples
  • The team that built the model does all the rating
Strong human evaluation
  • Two raters on a sample, answers shuffled and labels hidden
  • Specific questions with worked examples for each level
  • Domain experts and people who speak the users' languages
Watch out
Treat raters fairly. Pay them properly, give clear instructions and reasonable workloads, and warn them before showing disturbing content (for example, in safety evaluations). Tired or rushed raters give poor labels.
Try this. With a friend, each label the same 10 chatbot answers as good or bad, without talking. Count how many you agree on, then compute kappa with the function above.

Watch and read more

Lab

Two sets of labels on the same items, a kappa value, and an improved rubric.

~40 min

The problem

  1. Take 20 answers from your test set work (or any 20 chatbot answers you may use). Remove any sign of which model produced them and shuffle them.
  2. Write a short rubric with labels good, okay and bad, with one example of each.
  3. You and a friend label all 20 separately, without talking.
  4. Run the starter script to get raw agreement and kappa.
  5. Discuss every disagreement. Update the rubric, then both re-label the disagreed items and compute kappa again.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why can kappa be low even when raw agreement is high?Reveal

Kappa subtracts the agreement expected by chance. If both raters say "good" for most items, chance agreement is already high, so the room above chance is small. A few disagreements on the rare "bad" items then pull kappa down a lot. This is a feature: it shows the raters do not agree well on the cases that matter most.

Q2After fixing the rubric, kappa went up. Why should you be careful about computing it only on the items you discussed?Reveal

You and your friend now know each other's views on those exact items, so agreement on them is no longer independent. A fair check uses new items that neither of you has discussed. Keep some fresh items aside for the second round.

Please sign in to continue.

Questions people ask

Why hide which model produced which answer?

Because people are influenced by what they expect. If raters know an answer came from the new, expensive model, they may rate it more kindly. Blind, shuffled comparisons remove that influence.

Is human evaluation always better than an LLM judge?

Careful human evaluation by the right experts is usually the reference point. But untrained, rushed or non-expert raters can be worse than a well-calibrated judge. The quality comes from the process, not just from being human.

What if my labels have more than two categories, like good, okay, bad?

Cohen's kappa works with any number of categories, and the function above handles them. For ordered scales where "good vs okay" is a smaller disagreement than "good vs bad", a weighted kappa gives partial credit for near misses.

How many items should two raters both label?

Enough to see the pattern of disagreements, and enough that a single item does not swing the result a lot. Start with a sample, compute kappa, read the disagreements, and add more double-rated items if the result is unclear.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress