Inside the model: interpretabilityAdvancedLesson 3146 min read

Circuits: tracing a behaviour

Find which parts of a model cause a behaviour by swapping activations between runs. Induction heads are the classic worked example.

✦ Ask the tutor🧪 Lab · 35 min📝 Lesson test
Lesson in motion

In 60 seconds

Circuits: tracing a behaviour

Find which parts of a model cause a behaviour by swapping activations between runs. Induction heads are the classic worked example.

1/4
In simple words
A string of festival lights stops working. You swap one bulb at a time from a working string. When the lights come back on, you have found the part that mattered. Activation patching does this with the inside of a model.
A circuit is a small set of model parts (heads, MLP neurons or features) and the connections between them that together produce one behaviour. Finding circuits needs causal evidence: not "this part is active", but "changing this part changes the behaviour".

Activation patching

  1. 1

    Clean run

    Run a prompt where the model does the behaviour, for example completes "The Taj Mahal is in the city of" with "Agra". Save all activations.
  2. 2

    Corrupted run

    Run a similar prompt that breaks it, for example with a different monument. The answer changes.
  3. 3

    Patch

    Run the corrupted prompt again, but replace one component's activation with its value from the clean run.
  4. 4

    Measure

    How much of the gap between clean and corrupted output does that patch restore?
  5. 5

    Repeat

    Do this for every component. The ones that restore a lot are candidates for the circuit.
Patching is an intervention, so it answers the question probes cannot: does the model use this part here? A related method is ablation: set a component to zero or to its average value and see what breaks.

Patching a toy network

Activation patching on a two-layer toypython
import numpy as np

rng = np.random.default_rng(1)
W1 = rng.normal(size=(8, 6))        # input (6) -> hidden (8)
w2 = rng.normal(size=8)             # hidden (8) -> one output score

def run(x, patch=None):
    h = np.maximum(0, W1 @ x)       # hidden activations
    if patch is not None:           # patch = (unit index, value to force)
        i, v = patch
        h = h.copy()
        h[i] = v
    return w2 @ h, h

clean = rng.normal(size=6)          # e.g. a prompt where the model answers well
corrupt = clean.copy()
corrupt[2] += 3.0                   # change one "token" of the input

y_clean, h_clean = run(clean)
y_corrupt, _ = run(corrupt)
print(f"clean output {y_clean:.2f}, corrupted output {y_corrupt:.2f}")

# Patch one hidden unit at a time from the clean run into the corrupted run.
for i in range(8):
    y_patched, _ = run(corrupt, patch=(i, h_clean[i]))
    recovered = round((y_patched - y_corrupt) / (y_clean - y_corrupt), 3) + 0.0
    print(f"unit {i}: recovers {recovered:6.1%} of the gap")
Most units recover nothing, because the corruption did not change them. A few units carry the whole difference. In a real transformer you patch heads, MLP layers or SAE features at each position, and the effects are not always additive, because parts interact.

A classic example: induction heads

Language often repeats. If "Ravi Shankar" appeared earlier, then after the next "Ravi" a good guess is "Shankar". Induction heads are attention heads that do this: when the current token is A, look back for an earlier A, and copy the token that came after it.
  • They are usually described as a two-part circuit. An earlier previous-token head writes "the token before me was A" into each position. A later induction head then searches for positions whose previous token matches the current one, and copies what is there.
  • Researchers have found heads like this in many transformer models, from tiny ones to large ones.
  • In studies of training, they tend to appear fairly suddenly, around the same time the model gets better at using earlier context in the prompt. This links them to in-context learning, though they are not thought to be the whole story.
Note
Induction heads became a classic because the evidence came from several directions at once: attention patterns, patching and ablations, and the timing during training. One kind of evidence alone would have been much weaker.
Watch out
Circuits found for one narrow task in one model may not carry over. Large models often have backup components that take over when one is removed. A clean diagram is a simplified story of a messier computation.
Try this. Write a prompt with a repeated made-up name, for example "Zorvik Pallan met Mira. Later, Zorvik". What would an induction head predict next? Why is a made-up name a good test?

Watch and read more

Lab

Patching results on a toy network, and a test of what happens when two units back each other up.

~35 min

The problem

Use the code block from this lesson as your starter (needs numpy).

  1. Run it and list the units that recover most of the gap. Do the recovered shares add up to about 100%? Why might they, in this toy?
  2. Change the corruption to corrupt[4] += 3.0. Do different units matter now?
  3. Add redundancy: after creating W1, copy the row of the most important unit into another row (W1[j] = W1[i]) and set w2[j] = w2[i]. Patch each unit alone, then patch both together. What do you see?
  4. Write three sentences on what this means for finding circuits in large models.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your patching experiment shows head 5.3 restores most of the behaviour. A teammate concludes "head 5.3 is the circuit." What else should you check?Reveal

Whether the result holds on many prompt pairs, not one; whether other components matter when patched together (interactions and backups); what head 5.3 reads from and writes to, so you know its inputs and which later parts use its output; whether ablating it hurts the behaviour on new prompts; and whether the effect depends on your choice of corrupted prompt. A circuit is a path, not a single part.

Q2Why does the choice of corrupted prompt matter so much in patching?Reveal

Patching measures the effect of the difference between the two runs. If the corrupted prompt changes only the name of the monument, you find components that carry that fact. If it also changes grammar or length, you also find components for those, and mix up the story. Different corruptions answer different questions, so you must choose them to match the claim you want to test.

Please sign in to continue.

Questions people ask

What is the difference between patching and ablation?

Ablation removes a component's signal (sets it to zero or to its mean) and sees what breaks. Patching swaps in the activation from a different, carefully chosen run. Patching is more targeted, because the only difference between the runs is the thing you changed in the prompt.

Why do the clean and corrupted prompts need to be so similar?

So that the difference in output comes from one controlled change. If the prompts differ in length or in many words, many components will change for unrelated reasons, and the results become hard to read.

Isn't patching every component very slow?

It can be, since each patch needs a forward pass. Researchers use faster approximations based on gradients to shortlist components, then confirm the important ones with real patching. Approximations can be wrong, so confirmation matters.

Can I find induction heads in my tiny GPT?

Often, yes, if it has at least two layers and is trained on text with repetition. Feed it a random sequence of tokens repeated twice, and look for heads that, in the second copy, attend to the token right after the earlier match. Then ablate them and see if predicting the repeated half gets worse.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress