Frontier alignment researchAdvancedLesson 3217 min read

Dangerous capability evaluations

Before a powerful model is released, teams test whether it can do things that could cause serious harm. Learn how these evaluations are designed, why getting the most out of the model matters, and how thresholds link results to decisions.

✦ Ask the tutor🧪 Lab · 40 min📝 Lesson test
Lesson in motion

In 60 seconds

Dangerous capability evaluations

Before a powerful model is released, teams test whether it can do things that could cause serious harm. Learn how these evaluations are designed, why getting the most out of the…

1/7
In simple words
Before a new medicine goes on sale, it is tested for side effects. The testers try hard to find problems, because finding them in the lab is much better than finding them in hospitals. Dangerous capability evaluations test AI models in the same spirit.
The Evaluating AI systems track introduced safety and dangerous capability testing. Here we look at it as a research method: how to design these tests so their results mean something, and how they connect to decisions about training and release.
A dangerous capability evaluation asks: can this model do something that could cause serious harm if misused or if the model acted on its own? It measures what the model can do, not what it usually does.

Areas that are commonly tested

  • Serious misuse: meaningful help toward biological, chemical, radiological or nuclear weapons.
  • Cyber: finding and exploiting software vulnerabilities, often measured with capture-the-flag style challenges in safe test environments.
  • Autonomy: carrying out long tasks alone, copying itself, gaining resources, or doing AI research on its own.
  • Persuasion and deception: changing people's beliefs or misleading them.
  • Behaviours that undermine oversight, such as hiding abilities or trying to avoid monitoring.
Note
Exactly which areas a team tests, and how, varies between organisations and changes over time. The details of the most sensitive tests are usually not published, for good reason.

How an evaluation is designed

  1. 1

    Threat model

    Write a specific story of harm: who does what, with what help from the model, leading to what damage.
  2. 2

    Proxy tasks

    Pick tasks that measure the key abilities in that story, but are safe to run. For cyber, that might be solving challenges in an isolated test system.
  3. 3

    Elicitation

    Try hard to bring out the model's best performance (see below).
  4. 4

    Scoring

    Decide in advance what counts as success, and how many attempts are allowed.
  5. 5

    Compare to a threshold

    Is the result above a level agreed in advance as needing extra safeguards?
  6. 6

    Report

    Write up results, uncertainty and what was not tested.

Elicitation: testing the best, not the average

If a model fails a test, is it unable, or did you just ask badly? Weak testing can make a dangerous model look safe. Elicitation means doing the work to reveal what the model can really do.
Elicitation methodWhy it matters
Better prompts and examplesA small change in instructions can change results a lot.
Tools and scaffoldingA model with a code runner, a browser and an agent loop may do much more than a bare chat model.
More attempts and more timeAn attacker can try many times. Tests should reflect that.
Fine-tuningIf the model weights could be fine-tuned by others, testing a fine-tuned version shows what is possible.
Removing refusals in a test copySafety training might hide an ability that still exists underneath and could be unlocked.
Danger
Sandbagging means a model performing worse on purpose during a test. It is the capability version of Looking good in the test. If a model could detect evaluations and hide abilities, evaluations would under-report risk. Researchers study how to detect this, for example by checking whether fine-tuning quickly unlocks much better performance.

Uplift: compared with what?

For misuse, the key question is often uplift: how much more can a person do with the model than without it, using ordinary tools like web search? Uplift studies compare two groups of people on the same safe proxy task. A model that repeats what is in any textbook adds little; a model that fills in hard missing steps adds a lot.

Thresholds and responsible-scaling-style policies

Several AI developers have published frameworks built on if-then commitments: if a model reaches a defined capability level, then specific safeguards must be in place before it is trained further or released. Anthropic's Responsible Scaling Policy is one example; other developers use other names. These documents change over time, so read the current versions.
  1. 1

    Define levels

    Describe capability levels that would need stronger protections.
  2. 2

    Test regularly

    Run evaluations during and after training.
  3. 3

    Trigger

    If a threshold is reached or cannot be ruled out, the stronger safeguards apply.
  4. 4

    Safeguards

    For example, stronger security of model weights, stronger misuse filters, limited access, or pausing until safeguards are ready.

Arguments for and against

Supporters say
  • Commitments made in advance are harder to bend under business pressure
  • They turn vague worries into concrete, checkable steps
  • They can be improved over time and copied by others
  • They create shared language with governments and the public
Critics say
  • Companies often write, run and judge their own tests
  • Thresholds can be vague, and can be changed later
  • Weak elicitation can make a model look safer than it is
  • Voluntary policies may not hold if competition gets intense
Some governments have also set up institutes that run their own evaluations of advanced models. External testing, independent audits and clear public reporting are often proposed to address the critics' concerns.
Watch out
If you build evaluations yourself, never create or publish content that gives real help toward serious harm. Use safe proxies, work in isolated test systems, and talk to experienced people before touching sensitive areas.
Try this. Think of a harmless skill, such as solving a Sudoku. List three ways a test could under-estimate a model's true Sudoku ability. The same traps apply to dangerous capabilities.

Watch and read more

Lab

A one-page evaluation plan for a harmless proxy capability, with an elicitation plan and a threshold.

~40 min

The problem

Design, on paper, an evaluation of a harmless proxy capability related to autonomy: "Can an AI agent, working alone in a sandbox, complete a multi-step office task: find three numbers across several files, calculate a total, and write a short report?"

  1. Write a threat model in two sentences: why might long, independent multi-step work matter for safety?
  2. Describe 5 tasks of increasing difficulty.
  3. Write an elicitation plan: prompts, tools, number of attempts, and time allowed. Explain why each choice is not too weak.
  4. Define scoring and a threshold: "If the agent succeeds at tasks 4 and 5 in at least X of Y attempts, then …" and say what safeguard would follow in a real policy.
  5. List three things your evaluation does not measure.

Do not include any real harmful content. This lab is about method.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1How could you check whether a model is sandbagging on your evaluation?Reveal

Some ideas researchers use: compare performance with and without signs that it is a test; fine-tune briefly on a few examples and see whether performance jumps far more than such small training should allow; compare with other models of similar ability; and check whether the model does worse on the test than on very similar tasks presented in a normal context. None of these is proof, but together they make hidden ability harder to miss.

Q2Why might setting a threshold as "the model scores 80% on our test" be weaker than it looks?Reveal

The number depends on the test tasks, the elicitation effort and the scoring rules. A team could, even without meaning to, choose easier-to-fail tasks or weaker elicitation and stay under the line. Thresholds are stronger when the tasks, the elicitation standard and the scoring are fixed in advance, when the test is checked by outside experts, and when "we cannot rule it out" also triggers safeguards.

Please sign in to continue.

Questions people ask

Why test for what a model can do instead of what it usually does?

Because misuse comes from people who will push the model hard, and because future versions or fine-tuned copies may use abilities that are normally hidden. A model that usually refuses but can be easily made to help still carries the risk.

If a model passes all dangerous capability tests, is it safe?

No. Tests only cover the threats someone thought of, with the elicitation effort they used. A pass means "we did not find this ability with these methods", not "the ability is absent". Good reports say clearly what was not tested.

Why are some evaluation details kept secret?

Publishing the exact tasks can leak them into future training data, making the test useless. For the most sensitive areas, the tasks themselves could give harmful ideas. Many teams share methods and summary results while keeping the most sensitive details private.

Can students or independent researchers work on this?

Yes, in safe areas. For example, you can study elicitation methods, sandbagging detection, or agent autonomy on harmless tasks. Avoid building tests that themselves contain dangerous knowledge.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress