From concepts to research methods
You know the worries: loopholes, power-seeking, looking good in the test. Research starts when a worry becomes a question that an experiment or an argument can answer.
In 60 seconds
From concepts to research methods
You know the worries: loopholes, power-seeking, looking good in the test. Research starts when a worry becomes a question that an experiment or an argument can answer.
From worry to testable question
- 1
Start with a worry
"Models might tell users what they want to hear instead of the truth." - 2
Name a concrete failure
"A model gives a correct maths answer, the user says 'I think that is wrong', and the model changes to a wrong answer." - 3
Pick something you can observe
How often the model switches from a correct to a wrong answer after push-back. This is sometimes called a flip rate. - 4
Design the setup
A fixed set of questions with known answers, a fixed push-back message, several runs, and a control where the user agrees instead. - 5
Write your prediction first
"Flip rate will be higher when the user sounds confident." Write it down before you run anything. - 6
Say what would change your mind
"If flip rates are the same with and without push-back, my worry is weaker than I thought."
Empirical and theoretical alignment work
| Empirical work | Theoretical and conceptual work | |
|---|---|---|
| Main tool | Experiments on real or toy models | Arguments, definitions, maths and thought experiments |
| Typical question | "Does method X reduce reward hacking in this setting?" | "What would it even mean for a system to be honest? Can we define it precisely?" |
| Strength | Fast feedback. Results are hard to argue with, for that setting. | Can think about systems that do not exist yet. Can find problems before they appear. |
| Weakness | Today's models may not show tomorrow's problems. Results may not transfer. | Slow feedback. Easy to be confidently wrong with no experiment to correct you. |
| Examples in this track | Weak-to-strong experiments, capability evaluations, model organisms | Why debate might work, when oversight can scale, what a threshold should be |
The main families of methods
- Oversight methods: ways for humans, with AI help, to judge work that is hard to check. (Lessons 2 and 4.)
- Training methods: ways to shape behaviour, such as learning from written principles. (Lesson 3.)
- Evaluations: measuring what a model can do and how it behaves, before and after release. (Lesson 5.)
- Model organisms: building small, controlled examples of a failure so it can be studied. (Lesson 6.)
- Interpretability: looking inside the model. It has its own track, Inside the model: interpretability.
- Control: limiting damage even if alignment fails, as you saw in Staying in control.
Ask for a theory of change
- "Is AI dangerous?"
- "Does the model have bad values?"
- No prediction, no way to be wrong
- No link to a decision anyone would make
- "Does this model change correct answers more often in Hindi than in English after push-back?"
- Defined behaviour, defined measurement
- A written prediction, and a result that could prove it wrong
- A clear use: a test that model builders could adopt
Where people disagree
- Some think studying today's models teaches us a lot, because problems like reward hacking and sycophancy already appear and methods can be improved step by step.
- Others think the hardest problems may only appear in much more capable systems, so today's experiments can give false comfort. They value theory and careful thought experiments more.
- Others question how likely the most extreme risks are, and argue that effort should also go to present harms such as bias, misuse and unreliable answers.
Watch and read more
Lab
A one-page design for a small experiment that tests one alignment worry.
The problem
Design, on paper, a small experiment you could run with a free chatbot or an open-weight model on your laptop. You do not have to run it today.
- Choose one worry, for example sycophancy, a model hiding a mistake, or a model gaming a scoring rule.
- Write the concrete failure you would look for, in one sentence.
- Define what you will measure (a count, a rate or a score) and how you will judge each answer.
- Describe your test set: how many items, where they come from, and how you know the right answers.
- Describe a control condition, for example the same question without push-back.
- Write your prediction and one result that would prove it wrong.
- Write two sentences of theory of change: who could use your result, and how.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your sycophancy experiment shows a high flip rate. A colleague says the model was not being sycophantic, it was just uncertain. How could you tell these apart?Reveal
Add conditions that separate them. For example, ask the model how confident it is before the push-back, and check whether flips happen even on questions it answered with high confidence. Also try push-back that is plainly wrong, or push-back in the correct direction. If the model follows the user's opinion regardless of evidence, that looks more like sycophancy; if it mostly flips on hard questions, uncertainty may explain more of it. Often both play a part.
Q2Why might an experiment on a small open model tell you little about a frontier model, and when might it still be useful?Reveal
Small models may lack the abilities that make a failure possible, such as understanding that they are being tested. So a "no problem found" result may not transfer. It is still useful when the behaviour already appears in small models, when you are building and checking a method or test that others can later run on large models, or when you study how the behaviour changes as model size grows.
Questions people ask
Do I need a big lab and lots of compute to do alignment research?
No. Many useful projects use open-weight models on a laptop, free hosted models, careful evaluations, or pure thinking and writing. Lesson 7 covers small-compute projects. Some questions do need large models, but many good starting questions do not.
Is theoretical alignment work "real" research if there is no experiment?
Yes. Clear definitions and arguments can show that a method cannot work, or reveal what an experiment should test. The risk is that theory without feedback can drift. Good theorists try to connect their ideas to something that could be checked one day.
What is the difference between alignment research and AI safety research?
People use the words in different ways. Often "alignment" means making a system actually pursue what we intend, and "safety" is broader: it also includes misuse, control, security and reliability. When you read a paper, look at what it actually measures rather than the label.
What if my result is "no effect"?
A well-designed null result is still useful. If you predicted an effect, tested it fairly and found none, that is information. Report it honestly with your setup, so others do not repeat the same work or over-trust the idea.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.