Interpretability for safety: promise and limits
Many hope that looking inside models will one day catch deception or hidden goals. Here is what is real today, what is a hope, and how it fits with evaluations.
In 60 seconds
Interpretability for safety: promise and limits
Many hope that looking inside models will one day catch deception or hidden goals. Here is what is real today, what is a hope, and how it fits with evaluations.
Hopes, and where they stand
| Hope | The idea | Status today |
|---|---|---|
| Detect deception | Find signals that the model is saying something it represents as false. | Early research. Probes show some promise in controlled settings. Not reliable enough to depend on. |
| Find hidden goals | Audit a model for objectives it was not meant to have. | Researchers have run exercises where teams try to find a goal planted in a model on purpose. Useful practice; real hidden goals may be harder. |
| Check reasoning is honest | Compare the model's written reasoning with what its internals actually do. | An open problem. Written reasoning is not always a faithful record of the computation. |
| Monitor in deployment | Run cheap probes on activations to flag risky behaviour. | Possible for some narrow signals. Needs careful testing for missed cases and false alarms. |
| Remove dangerous knowledge | Find and remove specific capabilities. | Active research. Hard to show that knowledge is really gone, not just hidden. |
Why it is hard
- Scale: methods that work on small models or single tasks may not scale to very large models with countless features and circuits.
- Coverage: we can explain some behaviours in detail, but a tiny share of what a large model does. Safety needs to know about what we didn't find.
- Illusions: explanations can look convincing and be wrong. A feature label, a clean circuit diagram or a probe can mislead.
- Adversarial pressure: if you train a model to avoid triggering a deception probe, you may teach it to hide the signal instead of being honest. Detectors must not quietly become training targets.
- No ground truth: in real models we usually cannot check our explanation against a known answer, unlike the synthetic data in this track's labs.
How it combines with evaluations
- 1
Evals find behaviour
Dangerous-capability and safety evals show what the model can and will do. - 2
Interpretability asks why
For a worrying result, look for the features or circuits behind it. - 3
Interpretability suggests new tests
A found feature points to inputs that evals had not tried. - 4
Probes as extra monitors
Where they work, internal signals add a second line of defence next to output filters. - 5
Defence in depth
No single method is trusted alone. Evals, interpretability, monitoring and control all overlap.
- "We checked the internals, so the model is safe"
- "Our probe detects lying"
- "Interpretability will solve alignment soon"
- "Our checks found no sign of X with methods that have these known gaps"
- "On these test sets, this probe separates true from false statements; failure cases are listed"
- "Interpretability is one promising tool among several"
Watch and read more
Lab
A one-page review of one interpretability-for-safety claim, sorting evidence from hope.
The problem
This is a reading exercise. Pick one paper, blog post or news article about interpretability and safety (a search on arXiv for "sparse autoencoder" or "probing deception" is a good start). Then write one page with these headings:
- The claim in one sentence.
- The evidence: which methods (probe, SAE, patching, steering, evals), on which models, and on what data.
- What was causal and what was only correlation.
- Known gaps: scale, coverage, possible illusions, whether the model could hide the signal.
- How you would combine it with evaluations in a real safety process.
- A rewritten, careful one-sentence version of the claim.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Suppose a deception probe works well on a test set of statements the model was told to lie about. Why might it fail on a model that deceives for its own reasons?Reveal
The test set was built by instructing the model to lie, so the probe may detect "I was told to say something false" or the style of those prompts, not deception in general. A model pursuing its own goal might represent its deception differently, or not in the layers probed. Without real examples of spontaneous deception to test on, you cannot know how well the probe transfers. This is the ground-truth problem in a sharp form.
Q2Design a simple process that uses both evaluations and interpretability before deploying a new model. Where is each one strongest?Reveal
Run behavioural evals first: capability tests, safety tests and red-teaming. They are strongest at showing what the model does on many inputs. For any worrying or surprising result, use interpretability to look for the features or circuits behind it and to suggest new test inputs. Where probes have been validated, add them as monitors in deployment next to output filters. Keep detectors out of training. Write down what each method could not check. Interpretability is strongest at explaining and generating new hypotheses; evals are strongest at broad measurement. The decision should rest on all of them together.
Questions people ask
If interpretability cannot yet detect deception, why fund it?
Because behavioural tests have a hard limit: they only see outputs. If models get more capable and more aware of being tested, some check on the inside may become very valuable. Research takes years, so it has to start before it is urgently needed.
Could a model learn to hide from interpretability tools?
It is a real concern, especially if the tool's signal is used as a training objective. That is why many researchers argue to keep detectors separate from training, and to use several independent methods so that fooling one is not enough.
Is reading the model's chain-of-thought a form of interpretability?
It is a useful form of monitoring, and cheap. But the written reasoning is an output, and it is not guaranteed to reflect the real computation. Mechanistic methods try to check the internals directly. Both have a place.
How can I start contributing to this field?
Reproduce small results on open models, such as a probe, a toy superposition model or an SAE on a tiny GPT. Share your code and write up what did and did not work. Read recent papers on arXiv, and treat claims with the same care this track taught you.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.