TinyML: AI on tiny chipsMiddleLesson 2757 min read

Microcontrollers vs computers

A microcontroller has kilobytes of RAM, no operating system and a power budget in milliwatts. Learn how flash, RAM and power decide what model can fit.

✦ Ask the tutor 🧪 Lab · 20 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Microcontrollers vs computers

A microcontroller has kilobytes of RAM, no operating system and a power budget in milliwatts. Learn how flash, RAM and power decide what model can fit.

1/4

In simple words

Your school bag has a fixed size. You cannot carry the whole library; you pick the few books you need today, and you still need space for your lunch box. A microcontroller is a very small bag. The model is the book, and the working memory is the lunch box. Both must fit.

A laptop has gigabytes of RAM, a large disk, and an operating system that runs many programs at once. A microcontroller is much smaller. The numbers vary a lot by chip, so always check your board's datasheet. As a rough picture:

	Laptop or phone	Typical microcontroller
Working memory (RAM)	Gigabytes	Kilobytes to a few hundred kilobytes (some have a little more)
Storage for program and model	Hundreds of gigabytes	Flash memory: tens of kilobytes to a few megabytes
Operating system	Windows, Android, Linux, macOS	Often none: one program runs directly on the chip, or a very small real-time OS
Power	Watts	Milliwatts, and much less when sleeping
Maths	Fast floating point, often a GPU	Some chips have a floating-point unit, some do not; integer maths is usually cheaper

Note

A famous beginner board, the classic Arduino Uno, has only 2 KB of RAM. It is great for learning electronics, but too small for most ML. TinyML boards usually use more capable chips, for example Arm Cortex-M based chips or the ESP32 family. Check your board's current specifications.

Two kinds of memory

Flash keeps things when the power is off, like a notebook. Your program and the model's weights are stored here. Flash is larger than RAM, but it fills up.
RAM is working memory, like a desk while you study. It holds the input (for example the last second of audio), the numbers the model computes between layers (activations), and your program's variables. In TensorFlow Lite for Microcontrollers, this working area for the model is called the tensor arena.

A memory budget in four steps

1
Weights size
Number of weights × bytes per weight. float32 uses 4 bytes; int8 uses 1 byte.
2
Free flash
Total flash minus the program without the model. The weights must fit here.
3
Working memory
Input buffer + tensor arena + other variables. This must fit in RAM.
4
Leave some room
Keep a margin. Libraries, stack use and later features need space too.

Python: model size at 4 bytes and 1 byte per weightpython

# How big is a model? Weights x bytes per weight.
for params in [10_000, 100_000, 1_000_000, 1_000_000_000]:
    f32_kb = params * 4 / 1024
    i8_kb = params * 1 / 1024
    print(f"{params:>13,} weights: float32 {f32_kb:>12,.0f} KB   int8 {i8_kb:>12,.0f} KB")

Notice the last line: a model with a billion weights needs close to a gigabyte even at 1 byte per weight. That is why TinyML models are designed small from the start.

No operating system: what changes

There is usually no file system to "open model.tflite". The model is compiled into the program as an array of bytes.
Memory is often planned in advance. Many TinyML runtimes avoid asking for memory while running, so the program cannot run out of memory halfway.
You flash (upload) the whole program to the board, then read results through a serial connection, a light or a small display.
The chip spends most of its time asleep to save power, waking up on a timer or a sensor signal.

Power in milliwatts

Power is energy per second. A battery holds a fixed amount of energy, so the lower the average power, the longer it lasts. The trick is duty cycling: sleep most of the time, wake briefly to read the sensor and run the model, then sleep again. The average power is mostly decided by how long the chip is awake.

Real example

A student in Pune builds a band that detects when an elderly person may have fallen. The first version runs the model all the time and drains the battery quickly. The second version runs a very cheap check first, "did the motion suddenly change?", and only then runs the model. The model runs far less often, and the battery lasts much longer.

Watch out

Specifications on shopping sites can be wrong or mixed up between board versions. Before you design around a number, read the chip maker's datasheet for the exact part on your board.

Watch and read more

Lab

Three working Python functions that tell you whether a model fits on an example board.

~20 min

The problem

The starter describes an example board with 256 KB of RAM and 1024 KB of flash. The firmware already uses 300 KB of flash and 60 KB of RAM. Fill in the three functions:

model_kb(n_params, bytes_per_weight): the size of the weights in KB (1 KB = 1024 bytes).
audio_buffer_kb(seconds, sample_rate, bytes_per_sample): RAM needed to hold a piece of audio.
fits(n_params, bytes_per_weight, arena_kb): True only if the weights fit in the free flash and the arena fits in the free RAM.

Run it and look at the table. Which models fit as float32, and which only as int8? Then add the audio buffer to the medium model's arena: does it still fit?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

# Memory budget for a microcontroller board.
# These sizes are an example board, not a real product. Check your own board's datasheet.
RAM_KB = 256           # working memory (SRAM)
FLASH_KB = 1024        # storage for the program and the model
PROGRAM_FLASH_KB = 300 # your firmware without the model
OTHER_RAM_KB = 60      # stack, sensor buffers, variables

def model_kb(n_params, bytes_per_weight):
    """Size of the weights in KB (1 KB = 1024 bytes)."""
    raise NotImplementedError("Multiply the number of weights by bytes per weight, then divide by 1024")

def audio_buffer_kb(seconds, sample_rate, bytes_per_sample):
    """RAM needed to hold this much audio, in KB."""
    raise NotImplementedError("samples = seconds * sample_rate; bytes = samples * bytes_per_sample")

def fits(n_params, bytes_per_weight, arena_kb):
    """True only if the weights fit in the FREE flash and the arena fits in the FREE RAM."""
    raise NotImplementedError("Free flash = FLASH_KB - PROGRAM_FLASH_KB; free RAM = RAM_KB - OTHER_RAM_KB")

models = [("small", 50_000, 20), ("medium", 250_000, 60), ("large", 1_000_000, 150)]
for name, params, arena in models:
    for bpw, kind in [(4, "float32"), (1, "int8")]:
        print(f"{name:7} {kind:8} weights {model_kb(params, bpw):8.1f} KB  fits: {fits(params, bpw, arena)}")
print("1 s of 16 kHz, 16-bit audio:", audio_buffer_kb(1, 16000, 2), "KB")

Output

No output. Add print(...) to see values.

You are done when

model_kb multiplies weights by bytes per weight and divides by 1024
fits uses the free flash and free RAM, not the totals
The learner notices that the medium model fits only as int8
The learner checks whether an audio buffer still leaves room in RAM

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your model's weights fit in flash easily, but the board crashes when the model starts. What would you check first?Reveal

RAM. The weights may fit in flash, but the tensor arena, the input buffer and the rest of the program must all fit in RAM at the same time. Check the arena size your runtime reports as really needed, the stack size, and any large buffers such as audio. Reduce the input size, use a smaller model, or free other memory.

Q2Why is it risky to plan with 100% of the flash and RAM?Reveal

Other things also need space: libraries, the stack, debug code and future features. Some memory use only shows up at certain moments, such as a deep function call. A board that is full has no room to fix bugs later. Keep a clear margin and measure on the real board.

Please sign in to continue.

Questions people ask

Why not just use a Raspberry Pi for everything?

A Raspberry Pi is a full computer running Linux. It is easier to program and much more powerful, but it needs more power and takes time to boot. For a battery sensor that must run for months, a microcontroller is usually the better choice. For a camera project with mains power, a Pi can be easier.

What is the tensor arena?

It is a block of RAM that you set aside for the model's working memory: input, output and the in-between values. You choose its size in your program. Too small and the model cannot start; too big and your other code has no room. Tools can report how much was really used.

Does the model need to fit in RAM or in flash?

Usually the weights stay in flash and are read from there, and only the working memory must fit in RAM. This depends on the runtime and the chip, so check your tool's docs.

Why is integer maths preferred?

Many microcontrollers do integer maths faster and with less energy than floating-point maths, and some have no floating-point unit at all. Some chips also have special instructions that speed up int8 operations. That is one reason int8 models are common in TinyML.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Microcontrollers vs computers

Microcontrollers vs computers

Two kinds of memory

A memory budget in four steps

Weights size

Free flash

Working memory

Leave some room

No operating system: what changes

Power in milliwatts

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask