Inside the model: interpretabilityAdvancedLesson 3105 min read

Probing: what does a layer know?

Train a simple classifier on a layer's activations to see what information is there. Then learn why 'it is there' is not the same as 'the model uses it'.

✦ Ask the tutor🧪 Lab · 30 min📝 Lesson test
Lesson in motion

In 60 seconds

Probing: what does a layer know?

Train a simple classifier on a layer's activations to see what information is there. Then learn why 'it is there' is not the same as 'the model uses it'.

1/5
In simple words
Imagine you want to know if a friend noticed it was raining. You cannot read their mind. But if you can guess "yes, they noticed" just by looking at their wet umbrella, then the information was there. A probe is that kind of guess, made from a model's inner numbers.
A probe is a small, simple model trained to predict some property from a layer's activations. For example: run many sentences through a model, save the residual-stream vector at layer 6, and train a classifier to predict "is this sentence about sport?". If the probe does much better than chance on new sentences, the information is decodable from that layer.

Why keep the probe simple

We usually use a linear probe: logistic regression, just one weight vector and a bias. If a big, powerful probe is needed, the probe itself may be doing the work. A linear probe asks a sharper question: is the property stored as a direction in activation space? In many models, researchers have found that many properties can be read out this way, which suggests models often store concepts as roughly linear directions.
  1. 1

    Collect

    Run labelled examples through the model and save activations at one layer.
  2. 2

    Split

    Keep a test set the probe never trains on.
  3. 3

    Train

    Fit logistic regression on the training activations.
  4. 4

    Compare

    Check test accuracy against chance and against a control task.
  5. 5

    Repeat per layer

    Plot accuracy by layer to see where the information appears.

A probe in numpy

Real activations need a real model. Here we fake them: we hide a yes/no fact along one secret direction in 64-dimensional noise. The probe has to find that direction.
Logistic-regression probepython
import numpy as np

def train_probe(X, y, steps=500, lr=0.1):
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(steps):
        p = 1 / (1 + np.exp(-(X @ w + b)))      # sigmoid
        w -= lr * X.T @ (p - y) / len(y)        # gradient of log-loss
        b -= lr * np.mean(p - y)
    return w, b

def accuracy(X, y, w, b):
    return np.mean(((X @ w + b) > 0) == y)
In the lab you will run this on the fake activations. The probe should do well on the real labels, and its weight vector should point almost the same way as the secret direction.

Two traps: control tasks and correlation

  • The probe may learn the task itself. With many dimensions and few examples, even random labels can be fit on training data. Always test on held-out data, and run a control task, such as shuffled labels. A good probe does well on the real task and near chance on the control.
  • Decodable is not the same as used. A layer may contain information that the model never uses for its output. For example, a probe might read the sentence length from a layer, even if the model's answer does not depend on length.
Overclaiming from a probe
  • "The model understands sport"
  • "Layer 6 is where the model decides the topic"
  • Reporting only training accuracy
Careful claims
  • "Topic is linearly decodable from layer 6 activations"
  • "Removing this direction changes the output, so the model seems to use it"
  • Reporting test accuracy, a control task and a baseline
Note
To test use, you need an intervention: remove or change the direction the probe found, then see if the model's behaviour changes. We come back to this idea in the circuits and steering lessons.
Try this. Suppose a probe on layer 2 of a Hindi-English model can predict which language a word is in. Write one sentence that states this finding carefully, without claiming the model "knows" languages.

Watch and read more

Lab

A working numpy probe on synthetic activations, with a control task and a check of the direction it learned.

~30 min

The problem

Run the starter on your computer (needs numpy). Then:

  1. Record the test accuracy, the control accuracy and the cosine between the probe and the true direction.
  2. Change the signal strength from 1.5 to 0.3, then to 0.0. What happens to all three numbers?
  3. Set n = 100 (keep the 75/25 split by changing the slices) and compare training accuracy with test accuracy. Explain the gap.
  4. Write two sentences on what this toy can and cannot tell you about probing a real model.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your probe finds "is this text toxic?" in a model with high accuracy. A colleague says: "Great, the model has a toxicity detector inside." What is wrong with that claim?Reveal

The probe shows the information is decodable, not that the model has a dedicated detector or uses it. Toxic text may correlate with other things, like certain words or topics, which the probe could pick up instead. To make the stronger claim, you would need interventions: remove the direction and see whether toxicity-related behaviour changes, and test on examples where the correlated features are separated.

Q2Why might a property be decodable from almost every layer, and what does that mean for where it is 'computed'?Reveal

The residual stream carries information forward, so once a property is written in an early layer it can stay readable in later layers. Some properties are also easy to recompute from the input at any layer. So decodability across layers says little about where the computation happens. To locate it, look at where accuracy first rises sharply and then use patching to test which components cause it.

Please sign in to continue.

Questions people ask

Why not use a deep neural network as the probe?

Because a powerful probe can compute the property itself from messy inputs. Then high accuracy tells you about the probe, not the model. Simple probes keep the question honest. Some researchers do use small non-linear probes, but they report the difference carefully.

What is a control task?

A version of the task where the labels carry no real information about the input, for example shuffled labels. If your probe also scores well on that, it is memorising, not reading something the model stores.

Where do I get activations from a real model?

From an open-weight model, you can save the output of each block during a forward pass. Libraries for this exist in the Hugging Face and PyTorch world, and some are built for interpretability. Check their current docs, because the tools change often.

If probe accuracy rises from layer 2 to layer 8, what does that tell me?

That the information becomes easier to read out linearly in later layers. It does not tell you the model computes it there on purpose or uses it. It is a useful map of where to look next.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress