Robots and world modelsMiddleLesson 2628 min read

World models: learning how the world moves

A world model predicts what will happen next if the robot takes an action. It lets a system imagine before it acts. Learn the idea, the methods and the debate.

✦ Ask the tutor🧪 Lab · 30 min📝 Lesson test
Lesson in motion

In 60 seconds

World models: learning how the world moves

A world model predicts what will happen next if the robot takes an action. It lets a system imagine before it acts. Learn the idea, the methods and the debate.

1/5
In simple words
Before you throw a ball to a friend, you picture where it will go. If you imagine it going too far, you throw softer. You have a little model of the world in your head. Researchers want robots and AI to have one too.
A world model is a model that predicts how the world will change. In its simplest form it answers: "If things are like this now, and I do this action, what will things be like next?" People use such models all the time without noticing: when you pour water, you already expect where it will go.

Two ways to choose actions

Model-free
  • Learn directly: in this situation, do this
  • Fast to act once learned
  • Needs lots of practice
  • Cannot easily "think ahead" about a new goal
Model-based
  • First learn how the world responds to actions
  • Then plan by imagining different action sequences
  • Can reuse the same model for new goals
  • Plans are only as good as the model's predictions
The Q-learning robot in the last lesson was model-free: it learned values for actions, not how the world works. A model-based system learns the rules of the world first, then uses them to plan.

Planning inside a learned model

  1. 1

    Collect experience

    Act in the world, or watch it, and record what happened after each action.
  2. 2

    Learn the model

    Train a model to predict the next situation from the current one and an action.
  3. 3

    Imagine

    Try many possible action sequences inside the model, without moving at all.
  4. 4

    Pick and act

    Do the first part of the best imagined plan in the real world.
  5. 5

    Check and repeat

    See what really happened, fix the plan, and add the new data to improve the model.
The idea of planning a few steps ahead, acting, and re-planning is old and well used in engineering, where it is called model predictive control. What is newer is learning the model from data, including from raw camera images, instead of writing the physics equations by hand.

A tiny world model in Python

This program explores a grid at random, records what each move did, and uses those records as its world model. Then it plans a route using only the model, and finally checks the plan in the "real" world.
Learn a world model from experience, then plan inside itpython
import random
from collections import Counter, defaultdict, deque

# The real world (the robot cannot see this code, it can only try actions).
SIZE, WALLS = 4, {(1, 1), (2, 1)}
MOVES = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)}

def real_world(state, action):
    r, c = state[0] + MOVES[action][0], state[1] + MOVES[action][1]
    if not (0 <= r < SIZE and 0 <= c < SIZE) or (r, c) in WALLS:
        return state
    return (r, c)

# 1. Explore at random and record what happened: (state, action) -> next state.
random.seed(0)
seen = defaultdict(Counter)
s = (0, 0)
for _ in range(2000):
    a = random.choice(list(MOVES))
    s2 = real_world(s, a)
    seen[(s, a)][s2] += 1
    s = s2

# 2. The learned world model: predict the most common outcome we saw.
def model(state, action):
    outcomes = seen.get((state, action))
    return outcomes.most_common(1)[0][0] if outcomes else None  # None = "never tried"

# 3. Plan inside the model only (breadth-first search), no real moves needed.
def plan(start, goal):
    frontier, came_from = deque([start]), {start: None}
    while frontier:
        s = frontier.popleft()
        if s == goal:
            steps = []
            while came_from[s]:
                s, a = came_from[s]
                steps.append(a)
            return steps[::-1]
        for a in MOVES:
            nxt = model(s, a)
            if nxt is not None and nxt not in came_from:
                came_from[nxt] = (s, a)
                frontier.append(nxt)
    return None

actions = plan((0, 0), (3, 3))
print("Plan made inside the model:", actions)

# 4. Check the plan in the real world.
s = (0, 0)
for a in actions:
    s = real_world(s, a)
print("Real world ends at:", s)
Notice that once the model is learned, you can ask for a different goal and plan again, with no new practice. That is the main attraction of world models. Real ones must predict images, forces and uncertain outcomes, which is far harder than a grid.

Video prediction

One way to learn a world model from rich data is video prediction: show a model video frames and train it to predict the next frames, sometimes given the action taken. To predict well, the model has to capture something about how objects move, fall, and hide behind each other. Some researchers instead predict a compact summary of the next moment, rather than every pixel, because many pixel details (like the pattern on a curtain) do not matter for acting.
Note
Video generation models that make realistic clips are sometimes described as world models. A clip that looks real is not proof that the model gets physics right, or that it responds correctly to actions. Look for tests of prediction accuracy and usefulness for planning, not only good looks.

Why many researchers see world models as important for AGI

This is a view held by many researchers, not an established fact. The argument goes like this:
  • People and animals seem to learn a lot about the world by watching and interacting, long before they learn language.
  • A system that can predict the results of its actions can plan, imagine "what if", and avoid mistakes before making them.
  • Text alone may miss much of how the physical world works, as discussed in What is still missing.
Others argue that large models trained on text, images and video may already be learning useful world knowledge inside them, and that separate world models are not required. The honest answer is that this is an open research question.
Watch out
A world model can be confidently wrong. If a robot plans inside a model that is wrong about, say, how slippery a floor is, the plan may look perfect and still fail. Planners can also find "loopholes" in the model: action sequences that only work because the model has an error. Always check plans in the real world, carefully.
Try this. Before you drop a steel spoon and a paper napkin, predict which lands first and how each moves. Then try it. Where was your own world model right or wrong?

Watch and read more

Lab

Run the tiny world model, break it on purpose, and explain how a wrong model leads to a wrong plan.

~30 min

The problem

Copy the Python program from this lesson into worldmodel.py and run it with python3 worldmodel.py. It needs only standard Python.

  1. Run it. Write down the plan and check that the real world ends at (3, 3).
  2. Change the exploration from range(2000) to range(30). Run it. Does it still find a plan? If plan returns None, the next loop will crash; add a check that prints "no plan found" instead. Explain what happened in one sentence.
  3. Break the world. After the model is learned, add a new wall at (3, 1) to WALLS, just before step 4. Now the real world differs from the model. Where does the robot end up? Why?
  4. Write how you would fix it: what should the robot do when the real result differs from what the model predicted?
  5. On paper: for a robot that pours water into a glass, list three things its world model must predict, and one way each could be wrong.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why do small errors in a world model matter more when you plan many steps ahead?Reveal

Each imagined step starts from the previous imagined step, not from reality. A small error in step one changes the starting point of step two, and so on, so errors pile up, just like distribution shift in behaviour cloning. That is why many systems plan only a short way ahead, act, observe the real result, and re-plan.

Q2A planner finds a clever route that the world model says works, but the real robot fails. What might have happened?Reveal

The planner may have exploited a mistake in the model: a place where the model predicts something impossible, like passing through a wall it never saw. Optimising hard against an imperfect model tends to find exactly those errors. Defences include keeping plans close to situations the model has seen, measuring the model's uncertainty, and checking plans against reality step by step.

Please sign in to continue.

Questions people ask

Is a simulator a world model?

In a sense, yes: it predicts what happens next. The difference is that a simulator is usually written by people from physics equations, while a learned world model is trained from data. Some researchers hope learned models can capture effects that are hard to write by hand.

Why not always use a world model?

Learning an accurate model is hard, and errors add up when you imagine many steps ahead. Planning also takes computing time. For some tasks, a direct model-free policy is simpler and works better. Many systems mix both.

Do language models have world models?

This is debated. They clearly capture some facts about the world from text. How much they model physical cause and effect, and how reliable that is, is still being studied. Be careful with strong claims either way.

What does the robot do when its model has never seen a situation?

In the tiny example, the model returns "never tried" and the planner avoids that move. Real systems need similar honesty: a way to know when the model is unsure, so the robot can explore carefully or ask for help instead of trusting a guess.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress