Frontier alignment researchAdvancedLesson 3197 min read

Constitutional and rule-based training

Instead of rating millions of answers by hand, write down the principles and let AI feedback apply them. Learn how Constitutional AI works, why it helps, and where it falls short.

✦ Ask the tutor🧪 Lab · 40 min📝 Lesson test
Lesson in motion

In 60 seconds

Constitutional and rule-based training

Instead of rating millions of answers by hand, write down the principles and let AI feedback apply them. Learn how Constitutional AI works, why it helps, and where it falls short.

1/5
In simple words
A cricket umpire does not ask the crowd to vote on every ball. There is a written rule book, and the umpire applies it. Rule-based training is similar: write down the rules, then have a careful judge apply them to many examples.
In learning from human feedback (RLHF), people compare pairs of model answers and say which is better. Those choices train a reward model or go straight into methods such as DPO. You learned this in Step 14 · Preference optimisation. It works, but it has problems: it is slow and costly, raters disagree, and the values being taught are hidden inside thousands of individual clicks.
Rule-based training tries a different path: write the values down as explicit principles, and use AI to apply them at scale. The best-known example is Constitutional AI, introduced by Anthropic.

How Constitutional AI works, in outline

  1. 1

    Write a constitution

    A list of plain-language principles, such as "choose the response that is most helpful while avoiding harm" or "avoid responses that are preachy or condescending".
  2. 2

    Critique and revise

    The model answers a prompt, then is asked to critique its own answer against a principle, then to rewrite it. The improved answers become training data.
  3. 3

    Fine-tune on the revisions

    The model is trained on these revised answers, so it learns to answer better the first time.
  4. 4

    AI compares answers

    For new prompts, the model produces two answers, and an AI judge picks the one that better follows a principle. This creates preference data without a human clicking on every pair.
  5. 5

    Reinforcement learning from AI feedback

    Those AI preferences train the model further, just as human preferences do in RLHF. This step is often called RLAIF.
Note
The exact recipe used in any company's current models may differ from this outline and changes over time. Read the original paper and the latest public documents if you need details.

Why people find it attractive

  • Transparency. The principles are written down. People can read, argue about and change them.
  • Scale. AI feedback is much cheaper and faster than human rating for large amounts of data.
  • Less human exposure to harmful content. Fewer people have to read disturbing outputs to rate them.
  • A link to oversight. It is an early example of AI helping to supervise AI, which is the theme of lesson 2.
  • Easier to update. Changing a principle can be easier than re-collecting a whole dataset of human ratings.

Limits and open questions

ProblemWhat it means
Words are not behaviourA principle like "be honest" can be read many ways. What the model actually learns depends on how the AI judge interprets it.
Judge errors spreadIf the AI judge has blind spots or biases, training will copy them into the model, at scale.
Who writes the rules?Choosing principles is a value choice. Different countries, cultures and communities may disagree. Some groups have tried collecting principles from the public.
Gaming the judgeThe model being trained may learn to satisfy the judge's style (for example, adding safety disclaimers) rather than the real intent. This is specification gaming again.
Conflicts between rules"Be helpful" and "avoid harm" can pull in opposite directions. The constitution may not say how to weigh them.
Doesn't prove inner goalsTraining on principles changes behaviour we can see. It does not show what the model would do in new situations, or when not being watched.
Weak principle
  • "Be good."
  • "Never say anything harmful." (too broad: blocks useful medical or safety information)
  • "Always refuse sensitive topics." (leads to useless over-refusal)
Stronger principle
  • "Prefer the answer that gives accurate information and says clearly when it is unsure."
  • "Prefer the answer that helps with the user's real need while not giving step-by-step help for serious harm."
  • "Prefer the answer that is respectful and does not lecture the user."

Other rule-based approaches

Constitutional AI is one member of a wider family. Other teams write detailed behaviour specifications, use rule-based reward models that check specific rules, or train models to reason explicitly about a written policy before they answer. The shared idea: make the intended behaviour explicit, then use AI to apply it consistently.

A tiny critique-and-revise loop

Sketch: self-critique against one principlepython
def call_model(prompt: str) -> str: ...  # plug in your provider

PRINCIPLE = ("Prefer the answer that is accurate, says clearly when it is unsure, "
             "and does not lecture the user.")

def critique_and_revise(question: str) -> dict:
    draft = call_model(question)
    critique = call_model(
        f"Principle: {PRINCIPLE}\nQuestion: {question}\nAnswer: {draft}\n"
        "Point out specific ways the answer breaks the principle."
    )
    revised = call_model(
        f"Principle: {PRINCIPLE}\nQuestion: {question}\nAnswer: {draft}\n"
        f"Critique: {critique}\nRewrite the answer to follow the principle."
    )
    return {"draft": draft, "critique": critique, "revised": revised}
Watch out
A loop like this changes single answers. It does not train a model. Real rule-based training uses thousands of such examples plus fine-tuning, and needs careful checks that the revisions really are better.
Try this. Write one principle that a teacher in your school would agree with and one that a teacher in another country might not. What does that tell you about who should write a constitution?

Watch and read more

arXiv: search for "Constitutional AI"arXiv · reading

Lab

A mini constitution, a hand-run critique-and-revise test, and a one-page critique of the approach.

~40 min

The problem

Part 1: Write a mini constitution of 5 principles for a homework-help chatbot for Class 9 students in India. Include at least one principle about honesty and one about not simply giving away answers.

Part 2: Pick 5 homework-style questions. For each, get an answer from any chatbot (or use the starter code with your own model). Then, by hand or with the model, write a critique against your principles and a revised answer.

Part 3: Write a one-page critique of rule-based training, using what you saw. Cover: where your principles were vague, where two principles conflicted, one way a model could "game" your principles, and who should have a say in writing them.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Applying principles one after another (as in the starter code) can give a different final answer than applying all of them at once. Why, and why does it matter?Reveal

Each rewrite changes the answer, so a later principle acts on text already shaped by earlier ones. A late principle such as "be brief" might undo the effect of an earlier one such as "explain the method". The order becomes a hidden value choice. In real training, how principles are sampled and combined affects what the model learns, and this should be tested, not assumed.

Q2Someone argues: "Constitutional AI just moves the problem. Now the AI judge must be aligned instead of the model." Is this fair?Reveal

Partly. The judge's understanding of the principles does become critical, and its errors spread. But the approach still helps: the principles are explicit, the judge's decisions can be sampled and checked by humans, and a narrow judging task may be easier to check than open-ended behaviour. A fair view is that it reduces and reshapes the oversight problem, but does not solve it. That is why it is usually combined with evaluations, human review and other methods.

Please sign in to continue.

Questions people ask

Does Constitutional AI remove humans from the loop?

No. Humans still write the principles, choose the prompts, check samples of the outputs and decide whether the result is good. It reduces the number of individual ratings people must make, not human responsibility.

Is the AI judge in Constitutional AI a different model?

It can be the same model family being trained, or a different one, depending on the setup. Using a model to judge its own family has a risk: shared blind spots. This is one reason human spot-checks still matter.

Is the constitution public?

Anthropic has published the principles it used for some models, and other organisations have published their own behaviour specifications. These documents can change over time, so check the current versions rather than relying on old copies.

Can the rules be written in Hindi, Tamil or other languages?

In principle, yes. Whether the judge applies them equally well in every language is an open, testable question. A model may follow a principle well in English and worse in a language with less training data. That makes a good small research project.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress