Inside the model: interpretabilityAdvancedLesson 3136 min read

Sparse autoencoders

Learn a large dictionary of sparse features from a model's activations, so that each activation is explained by a few readable pieces.

✦ Ask the tutor 🧪 Lab · 40 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Sparse autoencoders

Learn a large dictionary of sparse features from a model's activations, so that each activation is explained by a few readable pieces.

1/4

In simple words

A thali has many dishes mixed on one plate. If you only see the whole plate, it is hard to say what is in it. A sparse autoencoder is like a list of all possible dishes, with the rule "use as few dishes as you can" to describe each plate.

If features are directions packed in superposition, we want a method that finds those directions without being told what to look for. A sparse autoencoder (SAE) is one such method. It is a small network trained on a model's activations, not on text. It learns a dictionary of many more directions than the layer has dimensions, and describes each activation with only a few of them.

How it works

1
Collect activations
Run lots of text through the model and save, for example, the residual stream at one layer.
2
Encode
Multiply by an encoder matrix and apply ReLU. This gives many feature values, most of them zero.
3
Decode
Add up decoder directions, weighted by the feature values, to rebuild the activation.
4
Train
Minimise reconstruction error plus a sparsity penalty (often an L1 penalty on the feature values).
5
Interpret
For each feature, look at the inputs that make it fire most strongly.

The central trade-off

If the sparsity penalty is...	Reconstruction	Features	Risk
Too weak	Very good	Many active at once, each mixed	Back to polysemantic mush
Balanced	Good	Few active, often readable	The sweet spot you search for
Too strong	Poor	Very few active	The SAE ignores real information

A tiny SAE, trained by hand

We make fake activations from 32 hidden true features in 16 dimensions (so they must be in superposition), then train an SAE with 64 dictionary features. Because the data is fake, we can check if the SAE found the true features. With a real model you cannot check this so easily.

Core SAE step (forward pass and loss)python

import numpy as np

def sae_forward(xb, W_enc, b_enc, W_dec, b_dec, l1):
    pre = (xb - b_dec) @ W_enc + b_enc
    f = np.maximum(0, pre)                     # sparse feature values
    x_hat = f @ W_dec + b_dec                  # rebuilt activation
    recon = np.mean(np.sum((x_hat - xb) ** 2, axis=1))
    sparsity = np.mean(np.sum(f, axis=1))      # L1 penalty
    return f, x_hat, recon + l1 * sparsity

The full training loop, with gradients written out by hand, is in the lab. On this toy it usually recovers every true feature. Real models are much harder.

Interpreting features

Look at the top activating examples. A feature might fire on Devanagari script, on phone numbers, or on code comments.
Check what the feature does: add its decoder direction to the activations and see how the model's output changes.
Researchers often use a language model to write a first label from the top examples, then test the label on new text. Labels can be wrong, so testing matters.

Limits to keep in mind

Dead features: some dictionary entries never fire on any input. They waste capacity. Training tricks exist to revive them.
Not all features are found: the SAE never rebuilds activations perfectly. The leftover error may contain important features the dictionary missed.
Features depend on choices: a bigger dictionary may split one feature ("sport") into several ("cricket", "football"). There is no single true list.
Labels are guesses: a feature that seems to mean "Mumbai" may also fire on other things. Top examples can mislead.
Cost: training SAEs on large models takes a lot of compute and data.

Note

Researchers have trained SAEs on several language models and found many features that people find readable, and some that change behaviour when turned up or down. This is one of the main tools in the field now. It is also an active area, and newer variants keep appearing, so check recent work before building on one version.

Try this. Why must the dictionary be larger than the layer? Write your answer using the word "superposition".

Watch and read more

Lab

A sparse autoencoder trained from scratch in numpy that recovers hidden features from synthetic activations.

~40 min

The problem

Run the starter (needs numpy, runs in seconds). Then:

Record how many true features were recovered, how many dictionary features are dead, and the variance explained.
Change l1 to 0.03 and to 1.0. For each, record active features per example, variance explained and features recovered. Describe the trade-off in two sentences.
Change m to 16 (the same as the layer size). Does it still recover all 32 true features? Why not?
The SAE uses about 3 features per example, but the data has fewer than 2 true features on per example. Look at pairs of decoder rows with high cosine and explain what is happening.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import numpy as np

rng = np.random.default_rng(0)
d, n_true, n = 16, 32, 20000

# Synthetic "activations": each is a sparse mix of 32 hidden true features in 16 dims.
true_dirs = rng.normal(size=(n_true, d))
true_dirs /= np.linalg.norm(true_dirs, axis=1, keepdims=True)
on = rng.random((n, n_true)) < 0.05                  # about 1-2 features on per example
X = (on * rng.random((n, n_true))) @ true_dirs

# Sparse autoencoder: 64 dictionary features, ReLU, L1 penalty on codes.
m, l1, lr = 64, 0.3, 0.3
W_enc = rng.normal(size=(d, m)) * 0.1
b_enc = np.zeros(m)
W_dec = W_enc.T.copy()                               # shape (m, d)
b_dec = np.zeros(d)

for step in range(5001):
    xb = X[rng.integers(0, n, 256)]
    pre = (xb - b_dec) @ W_enc + b_enc
    f = np.maximum(0, pre)                           # sparse codes
    x_hat = f @ W_dec + b_dec
    err = x_hat - xb

# Backpropagation, written out by hand.
    g_xhat = 2 * err / len(xb)
    g_Wdec = f.T @ g_xhat
    g_bdec = g_xhat.sum(axis=0)
    g_f = g_xhat @ W_dec.T + l1 / len(xb)
    g_pre = g_f * (pre > 0)
    g_Wenc = (xb - b_dec).T @ g_pre
    g_benc = g_pre.sum(axis=0)
    g_bdec -= (g_pre @ W_enc.T).sum(axis=0)

for p, g in ((W_enc, g_Wenc), (b_enc, g_benc), (W_dec, g_Wdec), (b_dec, g_bdec)):
        p -= lr * g
    W_dec /= np.linalg.norm(W_dec, axis=1, keepdims=True)   # keep decoder rows unit length

if step % 1000 == 0:
        mse = np.mean(np.sum(err**2, axis=1))
        active = np.mean(np.sum(f > 0, axis=1))
        print(f"step {step}: reconstruction error {mse:.4f}, active features {active:.1f}")

# How many true features did we recover? Compare decoder rows with the truth.
codes = np.maximum(0, (X - b_dec) @ W_enc + b_enc)
dead = np.sum(codes.max(axis=0) == 0)
best = (W_dec @ true_dirs.T).max(axis=0)             # best match for each true feature
print("true features recovered (cosine > 0.9):", np.sum(best > 0.9), "of", n_true)
print("dead dictionary features:", dead, "of", m)
x_hat = codes @ W_dec + b_dec
print("variance explained:", round(1 - np.var(X - x_hat) / np.var(X), 3))

Output

No output. Add print(...) to see values.

You are done when

The starter runs and the three final numbers are recorded
The l1 experiment shows the trade-off: weaker penalty gives better reconstruction but more active, less clean features; stronger penalty the reverse
With m = 16, you explain that 16 directions cannot each match one of 32 features, so features get mixed
You find near-duplicate decoder rows and explain that several dictionary entries can share one true feature

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1On the toy data you can check recovered features against the truth. On a real model you cannot. What would you use instead to judge an SAE?Reveal

Several indirect checks: how much of the activation it reconstructs, and how much the model's loss gets worse when you replace the real activations with the SAE's reconstruction; how sparse the codes are; how many features are dead; whether features are interpretable when people or models label them and the labels predict firing on new text; and whether changing a feature changes the model's behaviour as expected. None is perfect alone.

Q2Why might a bigger dictionary split "cricket" into "batting", "bowling" and "IPL" features, and what does this say about the idea of a true list of features?Reveal

With more entries, the SAE can lower its loss by giving finer patterns their own direction, because each finer feature is still sparse and rebuilds some inputs more cheaply. So the features you get depend on the dictionary size and penalty. This suggests features may be organised at many levels of detail, and there may be no single correct list. The dictionary is a useful lens, not the model's own ground truth.

Please sign in to continue.

Questions people ask

Is a sparse autoencoder the same as the autoencoders used for compression?

No, almost the opposite. A compression autoencoder squeezes data into fewer numbers. A sparse autoencoder expands it into many more numbers, but forces most of them to be zero. The goal is readability, not compression.

Why normalise the decoder directions?

Without it, the SAE can cheat the L1 penalty: make feature values tiny and decoder directions huge, which gives the same reconstruction with a smaller penalty. Keeping each decoder direction at length 1 blocks this trick.

How do I know an SAE feature is real and not an artefact?

You cannot be fully sure. Good signs: it fires on a clear, consistent set of inputs; a label written from some examples predicts firing on new examples; and changing the feature changes the model's output in the expected way. Features that pass all three are more trustworthy.

Can I train an SAE on my tiny GPT?

Yes, and it is a good exercise. Save the residual stream at one layer over your training text, then train the lab code on it. The tiny GPT may have fewer clean features, but you will learn the workflow.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sparse autoencoders

Sparse autoencoders

How it works

Collect activations

Encode

Decode

Train

Interpret

The central trade-off

A tiny SAE, trained by hand

Interpreting features

Limits to keep in mind

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask