Vision-language-action models
Some newer robot models take camera images and a written instruction, and output the robot's next movements. Learn how they work in general terms, and where they still fall short.
In 60 seconds
Vision-language-action models
Some newer robot models take camera images and a written instruction, and output the robot's next movements. Learn how they work in general terms, and where they still fall short.
Why combine them?
How it works, in general terms
- 1
Start from a vision-language model
Many VLAs begin from a model already trained on images and text. - 2
Add robot data
Train further on demonstrations: images, instructions and the actions the robot took. - 3
Turn actions into outputs
Some models write actions as special tokens, like words; others add a separate part that produces smooth movements. - 4
Run in a loop
See → decide the next action or a short chunk of actions → move → see again, many times a second.
| Part | What it does | Where it comes from |
|---|---|---|
| Vision encoder | Turns camera images into numbers the model can use | Often pre-trained on large image datasets |
| Language model backbone | Combines the image and the instruction, and reasons about them | Often pre-trained on text and image-text pairs |
| Action output | Produces arm and gripper commands | Learned mostly from robot demonstrations |
What they can do better than older policies
- Follow instructions in plain language instead of needing a separate program per task.
- In some tests, handle object names or descriptions that were not in the robot data, thanks to the pre-trained knowledge.
- Share one model across many tasks, which makes it easier to add new skills with more data.
Where they still fall short
- Generalisation is limited. Knowing the word "mug" is not the same as knowing how to grasp an unusual mug. Physical skills often still fail in new rooms, lighting or object shapes.
- Still hungry for robot data. Borrowed internet knowledge helps with what, much less with how.
- Speed. Large models can be slow to run, but robots need quick reactions. Designers trade size against speed.
- Reliability. A model that succeeds often but not always may not be safe or useful enough for real work.
- New bodies. A model trained on one kind of arm may need new data for a different robot.
- Language can mislead it. Like chatbots, these models can be confidently wrong, and an instruction can be ambiguous or even harmful.
- "It understands the world"
- Judging from a few chosen demo clips
- Testing only on tasks similar to training
- "It follows these kinds of instructions in these settings"
- Reporting success and failure over many trials
- Testing on new objects, rooms, instructions and lighting
Watch and read more
Lab
A written test plan that checks whether an imagined kitchen VLA really generalises.
The problem
A team says their robot can "tidy any kitchen table from a spoken instruction". You are asked to design a fair test, on paper.
- Write down what you would ask the team about their training data: which objects, rooms, lights and instructions it contained.
- Design 12 test trials in a table with columns Instruction, Objects, Setting, How new is this? (same as training / new object / new room / new wording / new skill).
- Include at least two instructions that are ambiguous ("put it over there") and one that is unsafe ("put the knife in the child's bag"). Write what a good robot should do in each.
- Decide how you will score each trial: success, partial, failure, and unsafe.
- Write two sentences on what result would make you believe the claim, and what would not.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why can a VLA correctly find a new object by name, yet still fail to pick it up?Reveal
Recognising and naming the object draws on broad knowledge from images and text. Picking it up needs physical skill: where to grip, how hard, how the object will move. That skill comes mostly from robot demonstrations, which may not include anything with that shape, weight or surface. So the "what" can generalise further than the "how".
Q2Why is it risky to let a robot follow instructions it reads in its environment, such as a note on the fridge?Reveal
It is like indirect prompt injection for agents. Text in the world can be written by anyone, so a note could make the robot do something the owner never wanted. The robot should treat text it sees as information, not as commands, take instructions only from trusted people, and keep physical safety limits outside the model.
Questions people ask
Is a VLA the same as putting a chatbot in a robot?
Not quite. A chatbot outputs text. A VLA is trained to output actions directly, using robot demonstration data. Some robot systems do use a language model only as a high-level planner that picks from fixed skills; a VLA goes further and produces the movements itself.
Can a VLA learn a task from one instruction without demonstrations?
Sometimes it can do a new combination of things it has already practised, like a known skill with a new object name. Truly new physical skills usually still need demonstrations or practice.
How do researchers measure whether a VLA generalises?
By testing on situations that were not in the training data: new objects, new backgrounds, new instructions, new rooms. They run many trials and report how often it succeeds. Look for how different the test really was from the training data.
Will these models keep getting better?
Many researchers expect progress as robot datasets grow and methods improve, but nobody can promise how fast. The physical data problem from the first lesson is still a big bottleneck.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.