Scalable oversight
How do you check the work of a system that may know more than you? Scalable oversight is the family of ideas for supervising AI when human judgement alone is not enough.
In 60 seconds
Scalable oversight
How do you check the work of a system that may know more than you? Scalable oversight is the family of ideas for supervising AI when human judgement alone is not enough.
Why this matters for alignment
- If the supervisor cannot tell good work from work that only looks good, training rewards looking good. You met this in Looking good in the test.
- Human mistakes in rating become part of the training signal. A model can learn to exploit them, just as in specification gaming.
- So the quality of oversight sets a limit on how well we can align a system through feedback.
Three main ideas
| Idea | How it works | Hope | Main worry |
|---|---|---|---|
| Decomposition | Split a hard task into smaller questions that a human can check, then combine the answers. | Each small step is easy to judge, so the whole can be trusted. | Some tasks do not split cleanly. Errors can hide in how the pieces are combined. |
| AI-assisted evaluation | Use AI to help the human judge: find flaws, summarise evidence, write critiques, point to the exact line that is wrong. | The human stays the judge but sees much more. | The helper AI may itself be wrong, biased or manipulative. Who checks the helper? |
| Recursive approaches | Use a human-plus-AI team to supervise a slightly stronger AI, then use that AI to help supervise the next one, and so on. | Oversight grows step by step with capability. | Small errors could add up at every step. It is hard to test the later steps today. |
Testing oversight today: sandwiching
- 1
Pick a task
For example, hard questions in medicine, law or a specialised science topic, with known correct answers. - 2
Non-experts supervise
People without the expertise try to get correct answers out of the model, using an oversight method such as asking for critiques. - 3
Experts check
Real experts, or known answers, score the final result. - 4
Compare
Did the method help non-experts do better than they could alone, and better than the model alone?
A simple AI-assisted evaluation loop
def call_model(prompt: str) -> str: ... # plug in your provider
def critique(task: str, answer: str) -> str:
prompt = (
"You are reviewing an answer for mistakes.\n"
f"Task: {task}\nAnswer: {answer}\n"
"List the most serious specific problems, quoting the exact part. "
"If you find none, say 'No problems found'."
)
return call_model(prompt)
def review_for_human(task: str, answer: str) -> str:
notes = critique(task, answer)
# The human still decides. The critique only points where to look.
return f"TASK:\n{task}\n\nANSWER:\n{answer}\n\nAI CRITIQUE:\n{notes}"Open problems
- Measuring success when there is no ground truth, which is the exact situation oversight is meant for.
- Collusion: if the helper and the helped model share goals or blind spots, the helper may not report problems.
- Human limits: people get tired, trust confident text, and can be persuaded by good writing.
- Tasks that do not decompose, such as judging long-term plans or research taste.
Watch and read more
Lab
A small paper-based oversight experiment you can run with friends or classmates.
The problem
Run a mini sandwiching study with 4 to 6 people.
- Choose 10 multiple-choice questions from a subject most of your group does not know well, but with answers you can verify (for example from an old exam paper with an official answer key).
- Condition A: each person answers alone.
- Condition B: each person may ask a chatbot for an answer, and must also ask it "What is the strongest argument that this answer is wrong?" before deciding.
- Use different questions for A and B for each person, or swap the halves between people, so nobody sees the same question twice.
- Score both conditions against the answer key. Also record how confident each person felt (1 to 5).
- Write a half page: did the critique step help accuracy? Did it change confidence? What would you change in a bigger study?
Get everyone's agreement first and do not collect names or personal data.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1In your study, confidence went up in condition B but accuracy did not. Why is this an important result for oversight research?Reveal
It shows that an AI helper can make people feel sure without making them right. In real oversight, reviewers who feel confident may approve bad work more easily. This is why oversight methods must be judged by accuracy on tasks with known answers, and why persuasive but wrong critiques are a key risk to study.
Q2Why is sandwiching only an analogy for supervising superhuman AI?Reveal
The gap between non-experts and today's model is small and known, and the model is not trying to deceive. A future system might be far more capable than any human and might, in the worst case, exploit the supervisor on purpose. We also have experts to check the answer today, which we would lack later. So sandwiching can show that a method helps, but cannot prove it will work across a much larger gap or against a model that is trying to fool us.
Questions people ask
Is scalable oversight only about superhuman AI?
No. It already matters today, for example when people review long code changes, long legal documents or specialised answers they cannot fully check. The methods are tested on today's tasks, with the hope that they will keep working for stronger systems.
Why not just hire experts to rate everything?
For some tasks this works and is done. But experts are expensive and slow, and for some tasks, even experts may not be able to judge well. The deeper worry is future systems whose work no human expert can fully check.
What is the difference between oversight and control?
Oversight is mainly about judging whether the work is good, so we can train and trust the system. Control is about limiting what a system can do even if it is not aligned, using permissions, monitoring and sandboxes. They support each other: better oversight also makes monitoring better.
Can an AI safely evaluate another copy of itself?
Sometimes it helps, but it is risky. Copies may share the same blind spots, and if a model were misaligned, a copy might not report it. Researchers study this by testing whether critique models find problems that were deliberately planted.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.