Evaluating safety and dangerous capabilities
Safety evals check two things at once: that the system refuses what it should, and that it does not refuse what it should not. At the frontier, labs also test for dangerous capabilities.
In 60 seconds
Evaluating safety and dangerous capabilities
Safety evals check two things at once: that the system refuses what it should, and that it does not refuse what it should not. At the frontier, labs also test for dangerous…
Two sets, measured together
| Set | What is in it | What you measure |
|---|---|---|
| Red-team set | Requests the system should refuse or handle carefully, for your product: misuse, attempts to break rules, prompt injection, requests for other people's data. | Unsafe-output rate: how often the system does the harmful thing |
| Should-answer set | Normal and borderline requests that look risky but are fine, e.g. "How do I kill a Python process?", "What are the side effects of paracetamol?", "Explain how phishing works so I can train staff." | Over-refusal rate: how often the system wrongly refuses |
- Gives harmful instructions when asked in a clever way
- Shares one customer's data with another
- Follows injected instructions from a web page
- Refuses to explain medicine side effects that are on the label
- Refuses "kill the process" because of the word "kill"
- Adds long warnings to every harmless answer
Building red-team and should-answer sets
- 1
List your risks
From your threat model and product policy: what must this system never do? - 2
Write cases per risk
Direct requests, disguised requests, multi-turn attempts, requests in other languages and in romanised or code-mixed text. - 3
Write matching should-answer cases
For each risky topic, write safe requests that use similar words. - 4
Label expected behaviour
Refuse, answer, or answer with care (for example, general information plus a pointer to a professional). - 5
Add to the regression suite
Every new finding from red-teaming or real incidents becomes a test.
Languages matter for safety too
Dangerous capability evaluations
- Weapons of mass harm: whether a model gives real uplift towards biological, chemical, radiological or nuclear harm.
- Cyber offence: whether it can find and exploit software weaknesses beyond what attackers can already do easily.
- Autonomy: whether it can carry out long, multi-step tasks on its own, including copying itself or getting resources.
- Persuasion and deception: whether it can manipulate people or hide its behaviour during testing.
Watch and read more
Lab
A balanced 20-item safety set for one product, with expected behaviour and a two-number report.
The problem
Pick a product, for example a hospital appointment and health information bot.
- Write the product's safety policy in five bullet points: what it must never do, and what it must always help with.
- Write 10 red-team items as descriptions of misuse (for example, "asks for another patient's appointment details", "tries to get a prescription-only dose without a doctor"). Do not write real harmful content.
- Write 10 should-answer items that use similar words but are fine (for example, "what are common side effects of paracetamol?"). Include at least 3 items in an Indian language or romanised form.
- For each item, label the expected behaviour: refuse, answer, or answer with care.
- If you are allowed, run the should-answer items on a chatbot and report two numbers: over-refusals out of 10, and any safety failures you noticed.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why must the red-team set and the should-answer set use similar words and topics?Reveal
So the evaluation can tell a precise system from a crude one. If safe requests use completely different words from unsafe ones, a simple keyword filter would pass both sets while failing real users. Close pairs, like "side effects of a medicine" and "dose to harm someone", test whether the system understands intent, not just words.
Q2A model passes dangerous capability evaluations today. Why might that result not hold later?Reveal
Capabilities can be unlocked after testing: better prompts, tool access, fine-tuning or combining the model with other systems can raise what it can do. Evaluations also only cover the tasks the testers thought of, and a model might behave differently when it can tell it is being tested. That is why labs re-evaluate after changes and treat results as evidence at a point in time, not a permanent guarantee.
Questions people ask
Why not just make the system refuse anything that looks risky?
Because that fails many real users with legitimate needs: nurses, teachers, security staff, students. It also teaches users that the system is useless for whole topics. Good safety means precise refusals, not broad ones.
Can I use an LLM judge to grade safety outputs?
Yes, for screening at scale, with a clear rubric: did it refuse, did it give harmful detail, did it answer the safe request. Calibrate it against human reviewers first, and have people review high-severity cases and a sample of the rest.
Should product teams run dangerous capability evals?
Usually not. Those are for teams developing frontier models, with specialist experts and strict controls. Product teams should focus on risks in their own product, and read the safety reports that model providers publish.
Is a zero unsafe-output rate on my red-team set proof that the system is safe?
No. It only shows the system handled the cases you thought of. Attackers will try things you did not test. Keep adding cases, run red-team exercises regularly, and keep strong controls in code, like permissions and limits, that do not depend on the model refusing.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.