Frontier alignment researchAdvancedLesson 3206 min read

Debate and weak-to-strong generalisation

Two research ideas for supervising stronger systems: let AIs argue while a human judges, and study whether a weak supervisor can train a stronger model well. Learn what these experiments can and cannot show.

✦ Ask the tutor 🧪 Lab · 40 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Debate and weak-to-strong generalisation

Two research ideas for supervising stronger systems: let AIs argue while a human judges, and study whether a weak supervisor can train a stronger model well. Learn what these…

1/6

In simple words

A judge in court does not know everything about medicine. But when two expert witnesses argue and point out each other's mistakes, the judge can often see who is right. Debate in AI uses the same idea.

Part 1: Debate as oversight

In debate, two AI systems take opposite sides of a question. They take turns making arguments and pointing out flaws in the other side. A human judge, who could not answer the question alone, decides who won. The winner is rewarded, so the systems are trained to argue well.

The key hope is an idea about truth: it may be easier to point out a lie than to defend one. If one debater makes a false claim, the other has a reason to find the weakest step and show it to the judge. If this holds, being honest becomes the winning strategy, even when the judge is weaker than both debaters.

1
A question
"Does this long proof have an error?" or "Which of these two answers about this 50-page report is correct?"
2
Two debaters
Each commits to an answer and argues for it, turn by turn.
3
Pointing to evidence
Debaters can quote short parts of the source that the judge can check directly.
4
A judge decides
The human (or a weaker model) picks the more convincing side.
5
Measure
On questions with known answers: does the judge pick the correct side more often with debate than without?

Why debate might fail

Persuasion is not truth. The judge may prefer the more confident, fluent or likeable side.
Obfuscated arguments. A dishonest debater might build an argument so long and tangled that the honest side cannot find the exact flaw in the time allowed.
Shared blind spots. If both debaters are copies of the same model, both may miss the same mistake.
Hard to test at the scale that matters. Today's experiments use tasks where humans can check answers, which is exactly not the future case.

Some published experiments have found that debate helped non-expert judges answer questions more accurately in certain setups, such as reading-comprehension tasks where only the debaters could see the full text. Other results were mixed. Treat debate as a promising but unproven idea.

Part 2: Weak-to-strong generalisation

Future humans may have to supervise AI that is stronger than them. We cannot run that experiment yet. So researchers study an analogy: use a small, weak model as the supervisor, and a larger, strong model as the student.

1
Train a weak supervisor
A small model learns a task from true labels. It makes many mistakes.
2
Weak labels
The weak supervisor labels a new dataset, mistakes included.
3
Train the strong student
A much stronger model is trained only on those weak labels.
4
Compare with a ceiling
Train the same strong model on true labels. This shows the best it could do.
5
Ask the question
Does the student only copy its teacher's mistakes, or does it do better than the teacher?

A useful measure is performance gap recovered (PGR). If the weak supervisor scores W, the strong ceiling scores C, and the student scores S, then PGR = (S − W) ÷ (C − W). A PGR of 0 means the student is no better than its teacher. A PGR of 1 means the student reached its full potential despite weak labels.

In some published experiments, strong students did better than their weak supervisors, but did not reach their full ceiling. Some extra training tricks helped in some settings. The hope is that strong models already "know" the right concept from pretraining, and weak labels only need to point them at it.

What these experiments can and cannot show

They can show

Whether a method helps a stronger model go beyond a weaker supervisor, on today's tasks
Which kinds of supervisor mistakes the student copies
How to compare oversight methods with a clear number

They cannot show

That the same method will work for humans supervising far smarter AI
What happens if the strong model is trying to deceive the supervisor
That a result on one task transfers to very different tasks

A key disanalogy

A small model's mistakes may be different in kind from human mistakes. And a future strong model might understand human errors very well, which makes it easier for it to imitate them. The analogy is useful for building methods, but researchers argue about how much it tells us about the real case.

Try this. The lab below is a tiny weak-to-strong experiment in pure Python. Before running it, predict: will the strong student copy the weak teacher's mistakes, or fix them?

Watch and read more

arXiv: search for "weak-to-strong generalization" and "AI safety via debate"arXiv · reading

Lab

A toy weak-to-strong experiment in pure Python, with a control condition and a short write-up.

~40 min

The problem

The starter code builds a simple classification task. A weak model can only see 3 of the 6 input features. A strong model sees all 6. You will compare:

the strong student trained on the weak model's labels (mistakes that follow a pattern), and
a control: the strong student trained on labels with the same number of mistakes, but placed at random.

Write your prediction for both PGR values before running.
Run the code (Python 3, no extra packages). Then run it again with 3 different values in random.seed(...).
Record accuracy and PGR for each run in a small table.
Write half a page: why do patterned mistakes and random mistakes behave so differently here? What is this toy missing compared to real weak-to-strong experiments with pretrained language models?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

# Toy weak-to-strong experiment. Standard library only.
import math, random

random.seed(0)
D = 6                                   # number of features
TRUE_W = [1.0, -2.0, 0.5, 1.5, -1.0, 2.0]

def make_data(n):
    xs = [[random.gauss(0, 1) for _ in range(D)] for _ in range(n)]
    ys = [1 if sum(w * v for w, v in zip(TRUE_W, x)) > 0 else 0 for x in xs]
    return xs, ys

def train(xs, ys, features, epochs=20, lr=0.01):
    """Logistic regression that may only look at some features."""
    w = [0.0] * D
    for _ in range(epochs):
        for x, y in zip(xs, ys):
            z = sum(w[i] * x[i] for i in features)
            p = 1 / (1 + math.exp(-max(-30, min(30, z))))
            for i in features:
                w[i] += lr * (y - p) * x[i]
    return lambda x: 1 if sum(w[i] * x[i] for i in features) > 0 else 0

def accuracy(model, xs, ys):
    return sum(model(x) == y for x, y in zip(xs, ys)) / len(ys)

ALL, FEW = list(range(D)), [0, 1, 5]     # the weak model sees only 3 features
train_x, train_y = make_data(2000)
test_x, test_y = make_data(2000)

weak = train(train_x, train_y, FEW)                  # weak supervisor
ceiling = train(train_x, train_y, ALL)               # strong model, true labels
weak_labels = [weak(x) for x in train_x]             # patterned mistakes
student = train(train_x, weak_labels, ALL)           # strong model, weak labels

# Control: the same number of wrong labels, but placed at random.
wrong = sum(a != b for a, b in zip(weak_labels, train_y))
flip = set(random.sample(range(len(train_y)), wrong))
noisy_labels = [1 - y if i in flip else y for i, y in enumerate(train_y)]
student_random = train(train_x, noisy_labels, ALL)

w_acc, c_acc = accuracy(weak, test_x, test_y), accuracy(ceiling, test_x, test_y)
for name, m in [("strong on weak labels", student),
                ("strong on random-noise labels", student_random)]:
    s_acc = accuracy(m, test_x, test_y)
    pgr = (s_acc - w_acc) / (c_acc - w_acc)
    print(f"{name:30s} acc={s_acc:.3f}  PGR={pgr:.2f}")
print(f"weak supervisor acc={w_acc:.3f}   strong ceiling acc={c_acc:.3f}")

Output

No output. Add print(...) to see values.

You are done when

Predictions were written before the code was run
Results are reported for at least 3 random seeds, not just one run
The write-up explains that the student can copy patterned mistakes exactly, while random mistakes tend to cancel out
The write-up names at least one thing the toy lacks, such as a pretrained model that already knows the concept

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1In this toy, the student trained on the weak model's labels barely beats its teacher. Real experiments with pretrained language models have sometimes shown students clearly beating their teachers. What explains the difference?Reveal

Here the student is a simple model that starts knowing nothing. The weak teacher's mistakes follow a clean pattern that the student can fit perfectly, so copying the teacher is the easiest way to match the labels. A large pretrained model may already represent the true concept from pretraining. The weak labels then act more like a hint that points to a concept it already has, so it can "correct" some teacher mistakes. That is the core hope of weak-to-strong generalisation, and it is also why results depend so much on what the student already knows.

Q2What does this toy suggest about which human mistakes are most dangerous when supervising strong AI?Reveal

Random, one-off mistakes may be averaged out. Systematic mistakes, where humans are wrong in a consistent, predictable way (for example, always preferring confident-sounding answers), can be learned and copied exactly, because matching them is rewarded. A capable model is also better at spotting such patterns. So oversight research should pay special attention to systematic human biases, not just the overall error rate.

Please sign in to continue.

Questions people ask

Has debate been used to train major commercial models?

Debate is mainly studied in research settings. Some ideas from it, such as asking models for critiques and counter-arguments, appear in wider practice. Check current papers for the state of the art, since this area changes quickly.

Why use a small model as the weak supervisor instead of real humans?

It is cheap, repeatable and gives lots of data. You can also measure exactly how weak the supervisor is. Human studies are more realistic but slower and noisier. Good research programmes use both.

What does a PGR above 1 mean?

The student did better than the strong model trained on true labels. This is unusual and should make you check your setup, for example for noise in the test set, a lucky seed, or a ceiling model that was not trained well.

Could debate make models better at manipulation?

That is a real concern. Training systems to win arguments could reward persuasive tricks. This is why debate studies measure whether judges become more accurate, not just whether one side wins, and why some researchers are cautious about it.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Debate and weak-to-strong generalisation

Debate and weak-to-strong generalisation

Part 1: Debate as oversight

A question

Two debaters

Pointing to evidence

A judge decides

Measure

Why debate might fail

Part 2: Weak-to-strong generalisation

Train a weak supervisor

Weak labels

Train the strong student

Compare with a ceiling

Ask the question

What these experiments can and cannot show

A key disanalogy

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask