Robots and world modelsMiddleLesson 2607 min read

Simulation and sim-to-real

Real robots are slow and costly to train. Simulators let a robot practise a huge number of times. The hard part is making what it learned work in the real world.

✦ Ask the tutor 🧪 Lab · 35 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Simulation and sim-to-real

Real robots are slow and costly to train. Simulators let a robot practise a huge number of times. The hard part is making what it learned work in the real world.

1/4

In simple words

A pilot practises in a flight simulator before flying a real plane. Crashing in the simulator costs nothing. Robots can practise in computer simulators too. But a simulator is never exactly like the real world, so the robot may be surprised when it leaves.

A simulator is a program that imitates physics: gravity, collisions, friction, joints and motors. It can also draw camera images of the scene. Inside a simulator, a robot can try a task many times, much faster than real time, on many computers at once, with nothing breaking.

Why simulate?

Speed: many copies of the robot can practise at the same time, faster than real time.
Safety: falls and crashes hurt nobody.
Cost: no motors wear out.
Perfect information: the simulator knows exactly where everything is, which makes rewards and labels easy.

Reinforcement learning in one page

Simulators are often used with reinforcement learning (RL). Instead of copying an expert, the robot learns by trial and error from a reward: a number that says how well it is doing.

Word	Meaning	Grid-world example
State	The situation right now	Which square the robot is on
Action	What the robot can do	Move up, down, left or right
Reward	A score after each action	+10 at the goal, -1 for every step
Policy	The rule for choosing actions	"In this square, move right"
Episode	One try, from start to end	From the start square until the goal

The robot must balance exploring (trying new actions to find out what they do) and exploiting (using what already works). A simple method called Q-learning keeps a table of how good each action is in each state, and nudges the numbers after every step:

The Q-learning update, in one linepython

# new estimate = old estimate + learning_rate * (what we just saw - old estimate)
Q[s, a] = Q[s, a] + alpha * (reward + gamma * best_Q_of_next_state - Q[s, a])

Here alpha is how fast to learn and gamma (between 0 and 1) is how much future rewards count. Real robots have far too many states for a table, so they use a neural network instead, but the idea of learning from reward is the same. You will run a full tiny example in the lab.

Watch out

Rewards are easy to get wrong. If you reward a cleaning robot for "no dirt seen by the camera", it might learn to look away from the dirt. This is the same loophole problem as in Specification gaming: finding the loophole.

The reality gap

A simulator is a simplified copy of the world. Friction, the softness of objects, the delays in motors, camera noise and lighting are never modelled exactly. A policy that is perfect in simulation may exploit tiny errors in the simulator and fail on the real robot. This difference is called the reality gap, and getting across it is called sim-to-real transfer.

Domain randomisation

A popular trick is domain randomisation. Instead of one simulator setting, you change the settings randomly in every episode: object weight, friction, colours, lighting, camera position, motor strength, small delays. The policy cannot rely on any one exact setting, so it learns something that works across many. The hope is that the real world looks like just one more variation.

1
Build the simulator
Model the robot, objects and scene as well as you reasonably can.
2
Randomise
Vary physics and visuals widely in each episode.
3
Train
Use RL, imitation, or both, over many episodes.
4
Test in the real world
Start slowly, in a safe area, with a way to stop the robot.
5
Close the loop
Measure what went wrong and improve the simulator or the randomisation.

Other ways to cross the gap

System identification: measure the real robot carefully (weights, friction, delays) and make the simulator match.
Fine-tuning on real data: train mostly in simulation, then adjust with a smaller amount of real-world practice or demonstrations.
Simpler inputs: give the policy things that look the same in sim and reality, such as joint angles, instead of raw images.

Note

Researchers have shown sim-to-real transfer working in some settings, especially for walking and balance. It tends to be harder for tasks with soft, deformable or messy objects, like cloth, food or liquids, because these are hard to simulate well.

Try this. Imagine a simulator of a vegetable market for a robot that picks tomatoes. List five things the simulator would likely get wrong compared with a real market.

Watch and read more

Lab

A tiny Python grid-world where a robot learns, by Q-learning, to reach a goal, plus two experiments about the reality gap.

~35 min

The problem

Save the starter code as gridbot.py and run it with python3 gridbot.py. It needs only standard Python. The robot learns by trial and error to reach the goal in a 4 x 4 grid with two walls. It should find a shortest path of 6 steps.

Run it and read the printed path. Draw it on paper.
Change range(500) to range(5). Does it still find a good path? Why or why not?
Change the step reward from -1.0 to 0.0. Run a few times with different seeds. What changes, and why? (Hint: gamma still makes the goal worth more when it is closer.)
Reality gap experiment. Train as normal. Then, only in the final walk, make the world "slippery": change the last loop so that with probability 0.2 the robot takes a random action instead. Does it still reach the goal? Is the path longer?
Randomisation experiment. Now make step slippery during training too. Does the learned path change? Write two sentences on how this is like domain randomisation.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import random

# A 4 x 4 grid. The robot starts top-left and must reach the goal bottom-right.
SIZE = 4
START, GOAL = (0, 0), (3, 3)
WALLS = {(1, 1), (2, 1)}          # cells the robot cannot enter
ACTIONS = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)}

def step(state, action):
    """The 'world': move one cell. Returns (next_state, reward, done)."""
    dr, dc = ACTIONS[action]
    r, c = state[0] + dr, state[1] + dc
    if not (0 <= r < SIZE and 0 <= c < SIZE) or (r, c) in WALLS:
        r, c = state                  # bumped an edge or a wall: stay put
    if (r, c) == GOAL:
        return (r, c), 10.0, True     # reached the goal
    return (r, c), -1.0, False        # every step costs a little

Q = {}                                # Q[(state, action)] -> learned value

def q(s, a):
    return Q.get((s, a), 0.0)

def choose(s, epsilon):
    if random.random() < epsilon:
        return random.choice(list(ACTIONS))       # explore: try something random
    best = max(q(s, a) for a in ACTIONS)
    return random.choice([a for a in ACTIONS if q(s, a) == best])  # exploit, break ties randomly

alpha, gamma, epsilon = 0.5, 0.9, 0.2  # learning rate, discount, exploration
random.seed(1)
for episode in range(500):
    s = START
    for _ in range(100):              # cap the length of one try
        a = choose(s, epsilon)
        s2, reward, done = step(s, a)
        target = reward if done else reward + gamma * max(q(s2, b) for b in ACTIONS)
        Q[(s, a)] = q(s, a) + alpha * (target - q(s, a))
        s = s2
        if done:
            break

# Now follow what was learned, with no random exploring.
s, path = START, [START]
while s != GOAL and len(path) < 20:
    s, _, _ = step(s, choose(s, epsilon=0.0))
    path.append(s)
print("Path:", path)
print("Steps:", len(path) - 1)

Output

No output. Add print(...) to see values.

You are done when

The program runs and prints a 6-step path from (0, 0) to (3, 3)
With very few episodes, you observed and explained a worse or incomplete path (not enough experience)
The slippery-world test is reported honestly, with whether the goal was still reached
Your two sentences link training in a varied world to handling surprises later, like domain randomisation

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why does the -1 reward for every step make the robot prefer short paths?Reveal

Every extra step subtracts 1 from the total reward, so a longer path to the same goal earns less. The Q-values learn this, and the greedy policy picks actions that lead to the goal in fewer steps. With a step reward of 0, the discount gamma still makes a reward reached sooner worth more, so short paths are usually still preferred, but the pull is weaker and learning can be slower.

Q2A policy is trained in a perfectly predictable grid, then the real grid turns out to be slippery. Why can it fail badly, even though the goal and walls did not move?Reveal

The policy only learned what to do on squares along its usual path, because it rarely needed anything else. When a slip pushes it to an unusual square, its values there may be poorly learned, so its choices can be poor. Also, a path that passes close to walls may be fine without slips but risky with them. This is the reality gap in miniature: the training world was simpler than the real one.

Please sign in to continue.

Questions people ask

Is a simulator the same as a video game?

They share a lot: both use physics engines and draw 3D scenes. But games only need to look right to a player. Robot simulators need the physics to be accurate enough that what works there also works on a real robot.

Why not do reinforcement learning directly on the real robot?

Sometimes people do, for simple tasks. But RL often needs a very large number of tries, many of them bad. On a real robot that takes a long time, wears out parts and can be unsafe. Simulation, demonstrations or both are used to cut down real-world practice.

Does domain randomisation always work?

No. If the real world has an effect the simulator never includes at all, random variation of other settings will not teach it. Too much randomisation can also make the task so hard that learning slows down. It is a useful tool, not a guarantee.

Can I try reinforcement learning without a robot?

Yes. The lab in this lesson runs a tiny grid-world robot in plain Python. Open-source simulators and RL toolkits also exist; check their current documentation to pick one that runs on your computer.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Simulation and sim-to-real

Simulation and sim-to-real

Why simulate?

Reinforcement learning in one page

The reality gap

Domain randomisation

Build the simulator

Randomise

Train

Test in the real world

Close the loop

Other ways to cross the gap

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask