Robots and world modelsMiddleLesson 2596 min read

Learning from demonstrations

One of the most common ways to teach a robot is to show it what to do. Learn how demonstrations are collected, what behaviour cloning is, and why copying goes wrong.

✦ Ask the tutor🧪 Lab · 25 min📝 Lesson test
Lesson in motion

In 60 seconds

Learning from demonstrations

One of the most common ways to teach a robot is to show it what to do. Learn how demonstrations are collected, what behaviour cloning is, and why copying goes wrong.

1/5
In simple words
When you learn to make rotis, you watch someone first. Then you try, and your first roti is a funny shape. Robots can also learn by watching and copying. But like you, they get into trouble when something happens that they never saw.
Imitation learning means learning a skill from examples of someone doing it, instead of from a written rulebook. In robotics, the examples are called demonstrations. Each one is a recording of what the robot saw and what action was taken, step by step.

How demonstration data is collected

MethodHow it worksGood and bad
TeleoperationA person controls the robot from a distance, with a joystick, a handheld controller, a copy of the arm or a VR headsetData matches the robot's own body; slow, needs trained people
Kinesthetic teachingA person gently moves the robot's arm by hand while it recordsSimple and direct; tiring, and the person's hand may appear in the camera view
Hand-held toolsA person does the task with a gripper tool that has a camera on itCan be done in many places; the robot's body still differs a little
Human videosRecordings of people doing tasksPlenty of them; no forces, different hands and viewpoints
Teleoperation is very common because the recording is in the robot's own terms: its own cameras and its own joint movements. The downside is cost. Each demonstration takes a person's time, and a useful skill may need many demonstrations.

Behaviour cloning: copy the expert

The simplest form of imitation learning is behaviour cloning. It turns robot learning into ordinary supervised learning, like the examples in How a computer learns from examples.
  1. 1

    Record

    Collect many pairs of (what the robot saw, what the expert did next).
  2. 2

    Train

    Train a neural network to predict the expert's action from the observation.
  3. 3

    Run

    On the real robot: take a camera image, ask the network for an action, do it, repeat many times a second.
  4. 4

    Check

    Test on new starting positions and objects, not just the ones in the recordings.
The trained network is called a policy: a function from what the robot observes to what it does. The idea is simple and often works surprisingly well when there are enough good demonstrations.

Why copying drifts: distribution shift

The expert almost never made big mistakes, so the data has almost no examples of recovering from mistakes. Now the robot runs its policy. It makes a small error: the gripper is a little to the left. This puts it in a situation that looks slightly different from anything in the data. So its next action is a bit worse, which leads to a stranger situation, and so on. Errors pile up.
This is called distribution shift (or compounding errors): the situations the robot meets while acting are different from the situations in its training data, because its own mistakes take it there.
Real example
A learner driver copies an instructor who always stays in the centre of the lane. The learner drifts slightly to the side. The instructor never drifted, so the learner has never seen how to come back. A small wobble turns into a big one.

Ways people reduce the problem

  • Recovery demonstrations: deliberately show the robot starting from bad positions and fixing them.
  • DAgger (dataset aggregation): let the robot act, record the situations it reaches, ask the expert what it should have done there, add that to the data, and train again.
  • More variety: collect demonstrations with different objects, positions, lights and people.
  • Predicting short chunks of actions rather than one tiny step at a time. Some methods do this to make movement smoother and more consistent.

Other problems with demonstrations

  • People are not consistent. Two experts may do the same task in different ways. A simple network may average them into a movement nobody did, like going between two paths around an obstacle and hitting it.
  • The robot only learns what it was shown. It does not know why the expert did something, so it cannot reason about a new goal.
  • Bad demonstrations teach bad habits. Data quality matters as much as data amount.
Weak demonstration data
  • Every demo starts from the same position
  • Only perfect runs are kept
  • One person, one table, one kind of cup
  • Test only on the training setups
Stronger demonstration data
  • Many starting positions and object placements
  • Includes corrections from small mistakes
  • Different people, rooms, lights and objects
  • Test on setups the robot never saw
Note
Collecting demonstrations from people must be done with their consent. If cameras record inside homes or workplaces, other people's faces and private items may be captured. Plan for privacy before you record.
Try this. Think of a skill you learned by copying: tying shoelaces, riding a cycle, cooking. When did you first get into a situation your teacher never showed you? What did you do?

Watch and read more

Lab

A paper experiment that shows distribution shift, plus a plan for a better demonstration dataset.

~25 min

The problem

Draw a 6 x 6 grid on paper. Put a start square at the bottom-left and a goal at the top-right. Draw a short wall in the middle.

  1. Be the expert. Draw one clean path from start to goal. Write it as a table: square → move (up, down, left, right). This is your demonstration.
  2. Be the robot. The robot only knows moves for squares in your table. Start one square to the right of the start. What does the robot do? Write down what happens.
  3. Now imagine each move has a small chance of slipping one square sideways. Roll a die for each move: on a 1, slip right. Follow your table. How often do you reach a square with no rule?
  4. Fix it. Add rows to the table for squares next to your path, showing how to get back. This is like recovery demonstrations or DAgger.
  5. Write three rules you would follow if you were collecting real demonstrations for a robot that picks up cups.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why does distribution shift get worse the longer a task is?Reveal

Each step has a small chance of error. In a long task there are many steps, so errors have more chances to happen and to build on each other. Once the robot is off course, later actions are made from situations the policy never trained on, so they are less reliable. That is why long, multi-step tasks are especially hard for pure behaviour cloning.

Q2A behaviour cloning policy scores well on a held-out test set of recorded demos, but fails on the real robot. How can both be true?Reveal

The held-out test checks: "given an observation the expert saw, does the policy predict the expert's action?" That only uses situations the expert reached. On the real robot, the policy's own small errors lead to new situations that are not in any recording. So accuracy on recorded data does not measure how well the robot recovers. You must test by actually running the policy.

Please sign in to continue.

Questions people ask

How many demonstrations does a robot need?

It depends a lot on the task, the robot and the method. Simple tasks in a fixed setting may need fairly few. Tasks with many objects and positions need many more. Rather than trusting a number, test the policy on new situations and see where it fails.

Is behaviour cloning the same as reinforcement learning?

No. Behaviour cloning copies an expert's actions and needs no reward. Reinforcement learning, covered in the next lesson, learns from rewards by trying things. The two are often combined: start by copying, then improve with practice.

Why not just write rules instead?

For simple, fixed tasks, hand-written rules and classic control work very well and are still widely used. Learning helps when the task has too much variety to write rules for, like grasping many different soft or oddly shaped objects.

What is teleoperation used for besides data?

People also teleoperate robots to do work directly, for example where it is unsafe for a person to be present. In demos, remember that a robot may be teleoperated rather than acting by itself.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress