Inside the model: interpretabilityAdvancedLesson 3156 min read

Steering models with activations

Add a chosen direction to a model's activations while it runs, and its behaviour shifts. Useful for research, and a reminder of how fragile safety training can be.

✦ Ask the tutor🧪 Lab · 30 min📝 Lesson test
Lesson in motion

In 60 seconds

Steering models with activations

Add a chosen direction to a model's activations while it runs, and its behaviour shifts. Useful for research, and a reminder of how fragile safety training can be.

1/5
In simple words
When you ride a bicycle, a small push on the handle turns you left or right. You do not rebuild the bicycle. Steering a model is a small push on its inner numbers while it is thinking, to turn its behaviour a little.
If a concept is stored as a direction, then adding more of that direction should turn the concept up. Activation steering does exactly this: during a forward pass, add a vector to the residual stream at some layer, and let the rest of the model run as usual. No retraining is needed.

Finding a steering direction

  1. 1

    Contrast pairs

    Collect prompts that differ in one way, for example formal and casual versions of the same message.
  2. 2

    Save activations

    Run both sets and save activations at one layer.
  3. 3

    Difference of means

    Subtract the average casual activation from the average formal one. That is your direction.
  4. 4

    Add it

    During generation, add alpha times the direction at that layer.
  5. 5

    Check

    Does the output change as intended? What else changed?
Other ways to get a direction include a probe's weight vector or an SAE feature's decoder direction. Researchers have shown in some models that steering can shift tone, topic, sentiment and other traits.

Steering a toy, and a side effect

Difference-of-means steeringpython
import numpy as np

rng = np.random.default_rng(2)
d = 32
formal = rng.normal(size=d); formal /= np.linalg.norm(formal)
topic = rng.normal(size=d); topic /= np.linalg.norm(topic)
topic -= 0.4 * formal                        # the two directions overlap a little

def fake_acts(n, is_formal):
    base = rng.normal(size=(n, d))
    return base + (2.0 if is_formal else -2.0) * formal

# 1. Find a steering direction: difference of the two averages.
steer = fake_acts(500, True).mean(axis=0) - fake_acts(500, False).mean(axis=0)
steer /= np.linalg.norm(steer)
print("cosine with the true 'formal' direction:", round(steer @ formal, 3))

# 2. Add it to new casual activations and watch two readouts.
casual = fake_acts(200, False)
for alpha in [0, 2, 4, 8]:
    steered = casual + alpha * steer
    print(f"alpha={alpha}: formality {np.mean(steered @ formal):6.2f}   "
          f"topic readout {np.mean(steered @ topic):6.2f}")
The formality readout rises steadily with alpha, as planned. But the topic readout also moves, because the two directions overlap. In a real model this is a side effect: you asked for a more formal reply and also shifted something else. Too large an alpha can also push activations far from anything seen in training, and the output may become strange or broken.

Uses and risks

UsesRisks
Test a probe or SAE feature: does changing it change behaviour?Side effects on other features that share space
Study how a behaviour is representedLoss of quality or fluency at large strengths
Cheap control over tone or style in research settingsWith open weights, anyone can steer against the model's safety behaviour
Explore possible safety tools, like turning down a harmful traitA steering result on a few prompts may not hold on others
Danger
Researchers have shown in some open-weight models that refusing harmful requests seems to depend largely on one direction. Removing that direction weakened the refusals. This is a strong reminder: safety behaviour learned in fine-tuning can be shallow, and open weights make it easy to edit. It is also why steering results are studied carefully before being used as a safety tool.
Note
Steering is also a test of understanding. If you claim "this direction means formality", then adding it should make outputs more formal on new prompts, without changing the topic. If it does not, your label is incomplete.
Try this. In the code, remove the line that makes topic overlap with formal. Predict how the topic readout will behave now, then run it.

Watch and read more

Lab

A toy steering experiment with a measured side effect, plus a short plan for testing steering safely on a real open model.

~30 min

The problem

Use the code block from this lesson as your starter (needs numpy).

  1. Run it and record both readouts for each alpha.
  2. Remove the overlap line (topic -= 0.4 * formal) and run again. What changes?
  3. Find a direction for steering that raises formality without moving the topic readout. Hint: remove from steer its component along topic. Check it works.
  4. Write a five-line plan for testing a formality steering vector on a small open-weight model you run yourself: which prompts, which strengths, and how you would measure side effects.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Removing one direction weakens refusals in some open models. What does this suggest about how refusal is learned, and what does it mean for people releasing open weights?Reveal

It suggests safety fine-tuning may add a fairly simple, low-dimensional behaviour on top of capabilities that stay in the model, rather than removing the capability. For releases, it means safety behaviour in the weights can be edited cheaply by anyone. So decisions about open release need to consider what the underlying model can do, not only how it behaves after safety training.

Q2Your steering vector makes replies more polite on 20 test prompts. Why is that not enough to call it a "politeness direction"?Reveal

Twenty prompts is a small sample. You also need to check side effects (did topic, length, language or accuracy change?), test on prompts unlike the ones used to build the vector, compare with a random direction of the same size as a control, and see whether the effect is smooth as strength grows. A vector can change politeness while mainly encoding something else, like "formal English" or "customer-service style".

Please sign in to continue.

Questions people ask

How is steering different from fine-tuning?

Fine-tuning changes the weights permanently through training. Steering changes the activations while the model runs, and only when you choose to apply it. Steering is cheap and reversible, but usually less reliable than training for lasting changes.

Which layer should I steer at?

There is no single answer. Researchers usually try several layers and strengths and measure the effect. Middle layers often work well for abstract traits, but it varies by model and by concept.

Can steering be used in a real product?

Some people explore it, but it needs the model's weights or internals, careful testing for side effects, and ongoing checks. For most products, prompting and fine-tuning remain simpler and better understood.

Does steering prove a direction is the model's concept?

It is good evidence that the direction affects that behaviour. It does not prove the direction is the only or the exact representation. Side effects often show that the direction mixes several things.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress