Build a mini robotMiddleLesson 2716 min read

Giving your robot eyes: a tiny vision model

A camera and a tiny image classifier can tell your robot which way the path turns. Train one on small synthetic images and learn why real-world data matters.

✦ Ask the tutor 🧪 Lab · 30 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Giving your robot eyes: a tiny vision model

A camera and a tiny image classifier can tell your robot which way the path turns. Train one on small synthetic images and learn why real-world data matters.

1/6

In simple words

A baby learns what a cat looks like by seeing many cats: big, small, black, white, sleeping, jumping. After a while, the baby can spot a cat it has never seen before. A tiny vision model learns the same way: we show it many small pictures with the right answer, and it learns the pattern.

IR sensors tell the robot only about the few millimetres of floor under them. A camera sees much more: the path ahead, a sign, a ball, a person. But a camera gives thousands of numbers (pixels), not a simple "yes" or "no". An image classifier is a model that turns those pixels into a label, such as "path turns left", "straight" or "turns right".

Where the camera and the model can live

Setup	How it works	Keep in mind
Phone as camera	A phone on the robot, or held above, sends images to a laptop; the laptop runs the model and sends commands to the robot	Easiest to start; needs a reliable link (Wi-Fi, Bluetooth or a cable), and adds delay
Small computer with a camera	A Raspberry Pi (not the Pico) with a camera module runs a small model on the robot	More power-hungry and costly than a microcontroller; much more capable
Microcontroller camera board	Some ESP32 boards come with a small camera; very tiny models can run on the board	Very little memory: images must be small and models tiny

Running a model on a tiny device is often called TinyML. The trick is to make everything small: low-resolution images (sometimes just a few dozen pixels on each side, often in grey), and a small model. Check the current documentation of your board for which tools and model sizes it supports.

How you build one

1
Decide the labels
Keep them few and clear: for example left, straight, right.
2
Collect images
From the robot's own camera, in the real room, in different light, at different positions.
3
Label them
Put each image in the right group. Mistakes here become mistakes in the model.
4
Train and test
Train on most images; test on images the model has never seen.
5
Shrink and deploy
Make the images and the model small enough for your device.
6
Keep a safe fallback
If the model is unsure, slow down or stop. Keep the distance sensor as a safety layer.

Why synthetic images first?

In the lab we make fake 8 × 8 pixel images with code: a light floor with a dark stripe that leans left, stays straight or leans right, plus random noise and brightness. This is quick and teaches the idea. But a model trained only on fake images usually does worse on real camera images, because real images have shadows, reflections, motion blur and a different camera. This is the same sim-to-real gap you met with the simulator.

Data that fails in the real world

All images from one room, at one time of day
Only perfect, centred examples
Test images that are copies of training images

Data that holds up better

Different rooms, lights, floors and times of day
Off-centre, blurry and partly hidden examples too
A separate test set collected on a different day

Watch out

Cameras and privacy. A robot camera can record people who did not agree to it. Point it at the floor or the track, keep images on your own device, do not share images of other people (especially children) without permission, and delete what you do not need.

Do this

Never let a vision model be the only thing stopping your robot from hitting someone. Models make mistakes, especially in new lighting. Keep the ultrasonic "stop if too close" rule as a separate safety layer that the model cannot switch off.

Try this. In the lab, after your model works, make the test images darker (lower brightness). Does the accuracy drop? What data would you add to fix it?

Watch and read more

scikit-learn documentationscikit-learn · reading

Lab

A tiny classifier that tells from an 8 × 8 camera picture whether the path leans left, goes straight or leans right, with at least 85% accuracy on new images.

~30 min

The problem

This runs in your browser. make_image makes a fake 8 × 8 grey camera picture of the floor: light floor, dark tape leaning left, straight or right, with noise and random brightness.

Run the starter. It shows some example images and says there is no model yet.
Train a classifier on X_train and y_train and store it in a variable named model. For example LogisticRegression(max_iter=1000) or KNeighborsClassifier() from scikit-learn. Each row of X_train is one image flattened to 64 numbers.
Print the accuracy on X_test. Goal: at least 0.85 on new images (the check makes its own).
Make darker test images (change the brightness range in a copy of make_image) and write what happens to the accuracy, and what you would do about it on a real robot.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import numpy as np
import matplotlib.pyplot as plt

LABELS = ["left", "straight", "right"]

def make_image(label, rng):
    img = np.full((8, 8), rng.uniform(0.6, 1.0))             # light floor, random brightness
    lean = {"left": -1, "straight": 0, "right": 1}[label] * rng.uniform(0.4, 0.8)
    bottom = rng.uniform(2.5, 4.5)                            # where the tape is at the bottom row
    for row in range(8):
        col = bottom + lean * (7 - row)                       # higher rows are further ahead
        for c in range(8):
            if abs(c - col) < 1.0:
                img[row, c] = rng.uniform(0.0, 0.25)          # dark tape
    return np.clip(img + rng.normal(0, 0.08, (8, 8)), 0, 1)

def make_set(n, seed):
    rng = np.random.default_rng(seed)
    y = [LABELS[i % 3] for i in range(n)]
    X = np.array([make_image(lab, rng).ravel() for lab in y])
    return X, np.array(y)

X_train, y_train = make_set(300, seed=1)
X_test, y_test = make_set(150, seed=2)

fig, axes = plt.subplots(1, 6, figsize=(9, 1.8))
for ax, img, lab in zip(axes, X_train[:6], y_train[:6]):
    ax.imshow(img.reshape(8, 8), cmap="gray", vmin=0, vmax=1); ax.set_title(lab); ax.axis("off")
plt.show()

model = None                          # TODO: train a classifier on X_train, y_train
if model is None:
    print("No model yet. Train one and store it in model.")
else:
    print("test accuracy:", round(float(np.mean(model.predict(X_test) == y_test)), 3))

Output

No output. Add print(...) to see values.

You are done when

The model reaches at least 85% accuracy on new images
Accuracy is measured on test images, not the training images
The darker-image experiment is done and its effect is reported honestly
The write-up names real-world data (other rooms, lights, floors) as the fix, not only a bigger model

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your classifier gets 99% on the synthetic test set. Why is that not a promise it will work on your robot's real camera?Reveal

The test images came from the same code as the training images, so they share the same "world": the same tape shape, noise and brightness range. A real camera adds shadows, reflections, blur, lens distortion and a different angle. The honest test is on real images from the robot, collected on a different day and place from the training images.

Q2The robot must decide 10 times a second, and your board has very little memory. What would you change about the model and the images?Reveal

Keep images tiny and in grey, as in the lab, and pick a small model: a linear classifier or a very small neural network. Fewer pixels and fewer model weights mean less memory and faster decisions. Then test that accuracy is still good enough. If it is not, move the model to a phone or laptop and accept the extra delay, with a local safety stop on the robot.

Please sign in to continue.

Questions people ask

Can I train a model without a powerful computer?

For tiny images and simple models like logistic regression, yes, a normal laptop or even the browser is enough. Larger vision models need more power, and many people use free online notebooks. Check the current limits and terms of any service you use.

How many images do I need?

It depends on how different the classes look and how varied the real world is. Start with some tens of images per class, test honestly on new images, and add more images of the cases where the model fails. More varied data usually helps more than more of the same.

Why not use a big vision-language model instead?

You can, for example on a laptop or through an online service, but it is slower, needs a network or a strong computer, and costs more. For a simple, fast decision like "left, straight or right", a tiny classifier is often better. Big models are useful for open questions, like "what is on the table?".

What does "the model is unsure" mean?

Many classifiers give a probability for each label. If the top label has a low probability, or two labels are close, the model is unsure. A robot can treat that as "slow down and look again" instead of acting on a guess.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Giving your robot eyes: a tiny vision model

Giving your robot eyes: a tiny vision model

Where the camera and the model can live

How you build one

Decide the labels

Collect images

Label them

Train and test

Shrink and deploy

Keep a safe fallback

Why synthetic images first?

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask