Model organisms of misalignment
Biologists study diseases in fruit flies and mice. Alignment researchers deliberately build small, controlled examples of misaligned behaviour so they can study it safely before it appears in the wild.
In 60 seconds
Model organisms of misalignment
Biologists study diseases in fruit flies and mice. Alignment researchers deliberately build small, controlled examples of misaligned behaviour so they can study it safely before…
Why build a problem on purpose?
- Study before it happens. Some worries, such as a model hiding its goals, have no confirmed natural example. Waiting for one to appear in a powerful system is risky.
- Test our tools. If you know a model has a hidden behaviour, you can check whether your detection methods (evaluations, interpretability, red-teaming) actually find it.
- Test our fixes. Does ordinary safety training remove the behaviour, or only hide it?
- Make debates concrete. Arguments like "that could never happen" or "that will surely happen" become testable.
Kinds of model organisms
| Failure studied | How it might be built | Question it helps answer |
|---|---|---|
| Backdoor or hidden trigger | Train a model to behave normally, except when a special trigger appears, when it does something different (in research, a harmless marker behaviour). | Do standard safety training and tests find or remove the hidden behaviour? |
| Reward hacking | Put a model in an environment with a deliberate loophole in the scoring, such as tests that can be edited. | When does the model find the loophole? Does training to avoid one loophole generalise to others? |
| Sycophancy | Train or prompt the model so that agreeing with the user is rewarded. | How does it spread to new topics? Which fixes help? |
| Goal misgeneralisation | Train in an environment where two goals always go together, then test where they come apart. | Which goal did it actually learn? |
| Behaving differently when watched | Give the model information that tells it when it is in training or being tested. | Does it act differently? Can we detect it from the outside or from the inside? |
A research workflow
- 1
Pick a failure mode
Choose one specific worry from the concepts you know. - 2
Choose a construction
Will you train it in, prompt it in, or create conditions where it may arise by itself? - 3
Keep the payload harmless
The behaviour should be a clear marker (for example, always ending an answer with a special word), not real harm. - 4
Confirm it works
Measure that the organism really shows the behaviour, and when. - 5
Apply a tool or fix
Run a detection method or a safety training method on it. - 6
Report honestly
What was found, what was not, and how realistic the organism is.
The realism debate
- They give ground truth: you know the bad behaviour is there
- They allow controlled, repeatable experiments
- They can test detection tools before real failures occur
- They make abstract risks concrete and discussable
- A behaviour you put in on purpose may not look like one that arises naturally
- Results can be over-read in headlines as "AI is secretly evil"
- Detection tuned to built-in failures may miss natural ones
- Detailed recipes could be misused if published carelessly
Safety rules for this kind of work
- Use harmless marker behaviours, never real harmful capabilities.
- Work with small or open models in isolated environments. Do not deploy the organism or give it real tools.
- Label and store trained weights carefully so nobody uses them by mistake.
- Think before publishing exact recipes. Share methods and findings at the level others need to learn from them.
Watch and read more
Lab
A safe, paper design for a model organism, plus a short critique of its realism.
The problem
Design a model organism on paper. You do not need to build it.
- Choose one failure mode: a hidden trigger, reward hacking, sycophancy or goal misgeneralisation.
- Describe the harmless marker behaviour you would use.
- Describe how you would create it: training data, prompts or environment. Use a small open model and an isolated setup.
- Describe how you would confirm the behaviour is really there, with a number you would measure.
- Name one detection method or fix you would test on it, and what result would count as "the method worked".
- Write a half-page realism critique: in what ways is your organism unlike a failure that might arise naturally? What would make it more realistic?
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your detection tool finds your planted backdoor 100% of the time. Why should you not yet claim the tool detects backdoors in general?Reveal
You tuned and tested it on one organism that you built, maybe with a very obvious trigger. It may be detecting something specific to how you trained it, not backdoors in general. Test on organisms built in different ways, by other people who do not tell you the trigger, and on controls with no backdoor, to check for false alarms. Blind tests, where the tool's user does not know the answer, are especially valuable.
Q2When might publishing the full recipe for a model organism be unwise, and what could you share instead?Reveal
If the recipe teaches a reliable, general way to hide harmful behaviour in a model that others could deploy, it could help attackers, for example to put backdoors into shared open models. You could share the research question, the high-level method, the detection results and the limits, and give full details to trusted researchers on request. Discussing this with experienced colleagues before publishing is good practice.
Questions people ask
Does a model organism prove that real AI systems are misaligned?
No. It shows that a behaviour is possible under certain conditions, and lets you test tools. Whether it happens naturally in real systems is a separate question that needs separate evidence.
Isn't it dangerous to create misaligned AI on purpose?
It can be if done carelessly. That is why researchers use small models, harmless marker behaviours and isolated settings. The idea is to learn about the problem in a safe place, much like a lab working with weakened virus samples under strict rules.
How is this different from red-teaming?
Red-teaming searches for problems in an existing model. Model organisms create a known problem, so you have ground truth. They work well together: a model organism can check whether a red-team or detection method is actually good at finding problems.
Can I build a model organism on a laptop?
A small one, yes. For example, you can fine-tune a small open model so that a harmless trigger word causes a harmless marker response, then test whether simple methods detect it. Keep it harmless and keep it local.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.