Why look inside a model
Testing what a model does tells you a lot, but not why it does it. Interpretability tries to read the mechanism itself.
In 60 seconds
Why look inside a model
Testing what a model does tells you a lot, but not why it does it. Interpretability tries to read the mechanism itself.
Four reasons to look inside
| Reason | What it looks like | Example |
|---|---|---|
| Debugging | Find why a model fails, not only that it fails. | A crop-disease classifier works in the lab but fails in the field. Is it looking at the leaf, or at the background soil? |
| Trust | Check that a right answer came from a sensible reason. | A loan model gives good results, but does it secretly rely on a person's pin code as a stand-in for caste or religion? |
| Safety | Look for dangerous knowledge or goals that tests may miss. | A model behaves well in every test. Is it behaving well because it is good, or because it can tell it is being tested? |
| Science | Learn how learning systems represent the world. | How does a model store the fact that Chennai is in Tamil Nadu? |
Behavioural testing vs mechanistic understanding
- Tells you what the model did on the inputs you tried
- Cannot see inputs you did not think of
- Can be fooled if the model acts differently when it senses a test
- Cheap, fast and works on any model through an API
- Aims to explain why the model did it
- Could, in principle, reveal behaviour you never triggered
- Looks at internal signals, which are harder to fake on purpose
- Needs the weights, lots of compute and careful work
What we can look at
- Weights: the fixed numbers learned in training. Same for every input.
- Activations: the vectors computed while the model reads one particular input. In a transformer, the most studied is the residual stream, the vector for each token that every block reads from and writes to.
- Attention patterns: which earlier tokens each head looks at.
- Gradients: how much the output would change if an input or activation changed a little.
The main tools in this track
- 1
Probing
Does a layer contain some information? Train a small classifier on its activations. - 2
Attention maps
Where does each head look? Useful, but often misread. - 3
Features and sparse autoencoders
Split activations into many simpler, more readable pieces. - 4
Circuits and patching
Change one internal part and see what changes in the output. This tests cause, not just correlation. - 5
Steering
Push the activations in a chosen direction and watch the behaviour change.
Watch and read more
Lab
A sorted list of questions about a model: which need a behavioural test and which need a look inside.
The problem
This is a paper exercise. For each question below, write behavioural, mechanistic or both, and one line on why.
- Does our Hindi-English translation model get dates right?
- Does our resume-screening model use the applicant's gender, even though gender is not an input column?
- Does the chatbot refuse to explain how to make a weapon?
- Would the chatbot still refuse if the request were written in a rare language we did not test?
- Where in the network does the model store the capital of each Indian state?
- Does the model answer differently when the prompt says "this is a test"?
Then pick one question marked mechanistic and describe, in three sentences, what you would want to look at inside the model.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1If a model passes every behavioural test you can think of, why might you still want mechanistic understanding?Reveal
Because your tests are a finite sample, and the model may behave differently on inputs you never tried or when it detects it is being tested. Understanding the mechanism could, in principle, reveal behaviour you never triggered. In practice this is still a hope for large models, so tests remain essential.
Q2Why is it hard to turn billions of learned weights into a human explanation?Reveal
The weights were not designed to be read. Training only rewards good predictions, so the model can spread one idea over many units and pack many ideas into one unit. A human explanation must compress this into a few concepts, and there is no guarantee the model's real concepts match ours. Later lessons (superposition, sparse autoencoders) deal with exactly this problem.
Questions people ask
If I wrote every line of the code, why don't I understand the model?
The code only sets up the shape of the computation. Training fills in the numbers, and those numbers decide the behaviour. It is like knowing the rules of cricket but not a particular bowler's plan for the next over.
Is interpretability the same as explainable AI (XAI)?
They overlap. Classic explainable AI often gives a simple summary of one decision, such as which input words mattered most. Mechanistic interpretability tries to reverse-engineer the actual computation inside the network. Both are useful for different jobs.
Can I do interpretability on a model I only reach through an API?
Mostly no. You need the activations, and usually the weights. That is why a lot of research uses open-weight models or small models trained for the study. Behavioural testing is what you can do through an API.
Is interpretability mainly for safety?
Safety is one strong motivation, but not the only one. Engineers use it to debug models, and scientists study it because it is a new kind of object to understand.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.