Inside the model: interpretabilityAdvancedLesson 3126 min read

Features and superposition

One neuron often responds to many unrelated things. A leading explanation is that models pack more features than they have dimensions.

✦ Ask the tutor 🧪 Lab · 30 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Features and superposition

One neuron often responds to many unrelated things. A leading explanation is that models pack more features than they have dimensions.

1/3

In simple words

A school has only 5 lockers but 20 students. If each student comes on a different day, they can share lockers and it mostly works. But if many come on the same day, things get mixed up. A model stores ideas the same way: more ideas than slots, as long as few are "in use" at once.

A feature is a property of the input that the model seems to represent, such as "this text is in Tamil", "this is a number" or "this word is inside a quotation". The first hope was that each neuron would be one feature. Sometimes that happens. But researchers have found that many neurons are polysemantic: one neuron fires for several unrelated things, say both cricket terms and legal language.

The superposition idea

A layer with 512 dimensions can hold 512 perfectly separate directions. But the world has far more than 512 useful features. The superposition hypothesis says models store more features than dimensions, as directions that are not quite at right angles. This works because most features are sparse: in any one sentence, only a few are active.

If features are rarely active together, their small overlaps rarely cause trouble.
The cost is interference: when two overlapping features are active at once, they can blur or create a false signal.
A side effect: each neuron (one axis) ends up partly aligned with many feature directions, so it looks polysemantic.

A 2-D toy: five features in two numbers

Put five feature directions around a circle, like the points of a star. Encode by adding up the directions of the active features. Decode by checking each direction, with a ReLU threshold that hides small overlaps.

Five features in two dimensionspython

import numpy as np

# 5 features squeezed into 2 dimensions: directions spread like a pentagon.
k = 5
angles = 2 * np.pi * np.arange(k) / k
W = np.stack([np.cos(angles), np.sin(angles)])   # shape (2, 5)
bias = -0.35                                     # ReLU threshold hides small overlaps

def encode(f):          # 5 features -> 2 numbers
    return W @ f

def decode(h):          # 2 numbers -> 5 feature guesses
    return np.maximum(0, W.T @ h + bias)

print("overlap between feature directions:")
print(np.round(W.T @ W, 2))

single = np.array([1.0, 0, 0, 0, 0])             # sparse: one feature on
print("one feature  ->", np.round(decode(encode(single)), 2))

two = np.array([1.0, 0, 1.0, 0, 0])              # two features on together
print("features 0+2 ->", np.round(decode(encode(two)), 2))

With one feature on, the decoder gets it right: feature 0 lights up and the others stay at zero. With features 0 and 2 on together, both real features vanish and a phantom feature 1 appears, because 1 sits between them. That is interference. It is rare only if features are rarely on together.

Situation	Superposition works?	Why
Features are sparse	Mostly yes	Overlaps rarely meet, and the threshold hides small ones
Features often co-occur	Poorly	Overlaps add up and create errors
Some features are very important	Model may give them their own direction	Errors on them cost too much

Note

Toy models like this one, with some training added, have been studied in detail by researchers. In those toys, superposition appears when features are sparse and disappears when they are dense. Whether large language models follow the same rules is still being studied, but the idea explains polysemantic neurons well.

Watch out

Superposition is a hypothesis with good support, not a settled law. Other things can also make neurons look polysemantic. Keep the idea, but hold it loosely.

Try this. Before running the code, predict what happens if you turn on features 0 and 1 (neighbours) instead of 0 and 2. Then run it and check.

Watch and read more

Lab

Measure how interference in a toy superposition model grows as features become less sparse.

~30 min

The problem

Run the starter (needs numpy). It encodes random inputs with the pentagon toy and counts how often decoding gets the on/off pattern exactly right.

Record the error rate for each sparsity level printed.
Change k to 3 and to 8 (still squeezed into 2 dimensions). How does the error change?
Change bias to 0.0 and to -0.8. Explain what the threshold trades off.
Write three sentences linking your results to polysemantic neurons in real models.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

Output

No output. Add print(...) to see values.

You are done when

The error rate is recorded and seen to rise as p_on rises
More features in the same 2 dimensions gives more errors at the same sparsity
The bias is explained as a trade-off: a stronger threshold hides overlaps but can also hide weak real features
The three sentences connect sparse, overlapping directions to neurons that respond to many things

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1If a model can pack more features than dimensions, why doesn't it pack an unlimited number?Reveal

Every extra feature adds overlap with the others, so interference grows. At some point the errors from collisions cost more loss than the new feature saves. How many features fit depends on how sparse and how important they are. Rare, less important features are the cheapest to pack in; frequent, important ones tend to get cleaner directions.

Q2Why is superposition a problem for probing and for single-neuron analysis in different ways?Reveal

Single-neuron analysis fails because each neuron mixes many features, so its activity has no single meaning. Probing still works when you know which label to look for, since it finds a direction, not a neuron. But probing only finds what you ask about. Superposition means there may be many features you never thought to probe, which is why unsupervised methods like sparse autoencoders are attractive.

Please sign in to continue.

Questions people ask

If neurons are polysemantic, why did people ever study single neurons?

Because some neurons really are clean, especially in image models, and single neurons are easy to inspect. Polysemanticity was found by looking at many neurons and noticing that most did not have a single meaning. That discovery is what pushed the field towards features as directions.

Is a feature the same as a probe direction?

Related, but not the same. A probe finds a direction that predicts a label you chose. A feature is something the model itself represents, which you may not have thought of. Sparse autoencoders (next lesson) try to find features without choosing labels first.

Why does sparsity matter so much?

Because interference only happens when overlapping features are active together. If each feature is on in only a small share of inputs, collisions are rare, and the model gains a lot of capacity for a small error.

Does this happen in the embeddings I built earlier?

Possibly. Embedding vectors for a big vocabulary in a small dimension must share space. Words that rarely appear together can safely overlap, which is one reason small embeddings still work well.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.