TinyML: AI on tiny chipsMiddleLesson 2808 min read

Deploying and testing on a board

Follow the path from a trained model to a program running on a microcontroller. Then test it on real data, and understand the power and privacy benefits.

✦ Ask the tutor 🧪 Lab · 45 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Deploying and testing on a board

Follow the path from a trained model to a program running on a microcontroller. Then test it on real data, and understand the power and privacy benefits.

1/4

In simple words

Writing a recipe at home is one thing. Cooking it in a hostel kitchen with a small stove and a few pots is another. You check that everything fits, you cook it for real, and you taste it. Deploying a model is taking your recipe to that small kitchen and tasting the result.

The deployment flow

1
Train on your computer
In Python or a TinyML platform, with your features and labels.
2
Convert and quantise
Turn the model into the runtime's format (for LiteRT, a .tflite file), usually full int8, using a representative dataset.
3
Check on the computer
Run the converted model on held-out data. Accuracy should be close to before.
4
Turn it into C
Store the model's bytes as a C array, so it is built into the program in flash.
5
Write the board program
Read the sensor, compute the same features, run the model, act on the result.
6
Flash and test on real data
Upload to the board, then test with real people, places and noise.

Platforms such as Edge Impulse can do most of these steps for you and export a ready library or firmware for supported boards. Doing them by hand once helps you understand what those tools do. Either way, check the current docs: file names, commands and APIs change between versions.

The model as a C array

With no file system, the model file is pasted into the program as bytes. A command-line tool such as xxd -i can do this. The Python below shows the idea with a few bytes.

Python: turning model bytes into a C arraypython

# What "convert the model to a C array" means: the model file is just bytes.
# Here we use a few made-up bytes instead of a real .tflite file.
model_bytes = bytes([0x1C, 0x00, 0x00, 0x00, 0x54, 0x46, 0x4C, 0x33, 0x14, 0x00, 0x20, 0x00])

lines = []
for i in range(0, len(model_bytes), 8):
    chunk = ", ".join(f"0x{b:02x}" for b in model_bytes[i:i + 8])
    lines.append("  " + chunk + ",")
print("alignas(16) const unsigned char g_model_data[] = {")
print("\n".join(lines))
print("};")
print(f"const unsigned int g_model_data_len = {len(model_bytes)};")

A sketch of the board program

This is a simplified C++ sketch in the style of TensorFlow Lite for Microcontrollers. It is for a real board, not for the browser. Function names and headers change between versions, so start from the current official examples for your board.

C++ (for the board, not runnable here): run one window through the modelcpp

#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "model_data.h"   // g_model_data: the C array made from the .tflite file

constexpr int kArenaSize = 20 * 1024;           // working memory; measure and adjust
alignas(16) static uint8_t tensor_arena[kArenaSize];

static tflite::MicroMutableOpResolver<3> resolver;  // only the layer types the model uses
static tflite::MicroInterpreter* interpreter = nullptr;

void setup_model() {
  const tflite::Model* model = tflite::GetModel(g_model_data);
  resolver.AddFullyConnected();
  resolver.AddRelu();
  resolver.AddSoftmax();
  static tflite::MicroInterpreter interp(model, resolver, tensor_arena, kArenaSize);
  interpreter = &interp;
  interpreter->AllocateTensors();                // fails if the arena is too small
}

int classify(const float* features, int n) {
  TfLiteTensor* input = interpreter->input(0);
  for (int i = 0; i < n; i++) {                  // float feature -> int8, same scale as training
    int q = (int)roundf(features[i] / input->params.scale) + input->params.zero_point;
    input->data.int8[i] = (int8_t)(q < -128 ? -128 : (q > 127 ? 127 : q));
  }
  if (interpreter->Invoke() != kTfLiteOk) return -1;
  TfLiteTensor* output = interpreter->output(0);
  int best = 0;
  for (int c = 1; c < output->dims->data[1]; c++)
    if (output->data.int8[c] > output->data.int8[best]) best = c;
  return best;                                   // index of the most likely class
}

Note

Look at the quantisation step in classify: the board converts each float feature to int8 using the input's scale and zero point. This is the lesson Shrinking it to fit in action. If the features are computed differently from training, the model gets inputs it never saw.

Testing on the real board

Same input, same answer. Record a few windows on the board, run them through the model on your computer too, and compare the outputs.
Real people and places. Test with people who were not in the training data, in the real rooms, fields or machines.
Long runs. Leave it running for hours or days and count false alarms. Check that it does not crash or slow down over time.
Memory and timing. Check the arena size really needed, and how long one run of the model takes. It must finish before the next window arrives.
Edge cases. Low battery, a loose sensor, a new phone case, very hot or cold days.

Power and privacy benefits

Benefit	Why	How to keep it
Long battery life	The chip sleeps most of the time and sends only short results	Duty cycle; run the model only when a cheap check says something may be happening
Privacy	Raw audio, motion or images can stay on the device	Do not upload raw data by default; if you must collect data for improvement, ask clearly and keep it minimal
Works offline	No network is needed to decide	Plan how results are shown or stored when there is no connection
Fast response	No round trip to a server	Keep the model small enough to finish well within one window

Watch out

On-device does not mean automatically private or secure. Someone with the board may read the firmware and the model. Results you do send still say something about people. Protect any data you send, and be honest with users about what the device detects.

Try this. Read the C++ sketch and find three things that must match your Python training exactly. (Hint: think about features, input scale and the order of the classes.)

Watch and read more

TensorFlow Lite for Microcontrollers (LiteRT for Microcontrollers) source code and examplesGitHub · reading Edge Impulse documentationEdge Impulse · reading

Lab

A deployment and test plan for one TinyML project, and, if you have a board, a model running on it.

~45 min

The problem

Choose one idea from your idea cards in the first lesson, or the activity model from this track.

Budget: using your functions from Microcontrollers vs computers, check that the int8 weights and the working memory fit on a board you could use. Write the numbers and where you got the board's specifications.
Pipeline: list each step from trained model to flashed board, naming the tool you would use for each. Check the tool's current docs.
Match check: describe how you will prove the board computes the same features and scores as your Python code.
Field test: write a test plan: who, where, for how long, and how you will count false alarms per hour and misses.
Power and privacy: say when the chip sleeps, what data leaves the device, and what users are told.

If you have a supported board: follow its official TinyML example (for instance a gesture or keyword example), get it running, then replace the model with your own and run your test plan.

You are done when

The memory budget uses real specifications from a datasheet or official page, with a margin
Each step of the pipeline names a tool or command, with a note to check current docs
The plan compares board outputs with computer outputs on the same recorded inputs
The field test counts false alarms per hour and misses, with people not in the training data

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your model scores well in Python after int8 conversion, but on the board every window gets the same class. List three likely causes.Reveal

First, the input is not quantised the way the model expects: a wrong scale or zero point, or floats written into an int8 tensor. Second, the features differ: different units, axis order, window length or sampling rate. Third, the input buffer is not being filled (for example the sensor is not started, so every window is zeros). Record one window on the board, print its features and input tensor, and compare with Python step by step.

Q2A user asks: "Does your always-listening device record me?" How would you design it so you can honestly answer no?Reveal

Keep audio only in a short rolling buffer in RAM that is overwritten all the time, and never write it to flash or send it. Send only the event ("wake word detected") and nothing else by default. If you want recordings to improve the model, make it a separate, clearly explained opt-in. Then explain this simply to users, and make sure the firmware really does what you say.

Please sign in to continue.

Questions people ask

Which board should I buy first?

Pick a board your chosen tool officially supports, with the sensor you need built in (for example a motion sensor or a microphone), and good examples. Many maker boards qualify. Check the tool's current list of supported boards before buying.

What if the model does not fit on my board?

Work back through the track: use fewer features or a smaller model, make sure it is fully int8, shrink input buffers, and include only the operations the model uses. If it still does not fit, a board with more memory may be the honest answer.

How do I see what the model is doing on the board?

Print scores over the serial connection to your computer while you test. Many boards also have an LED you can light for each class. Remove heavy debug printing in the final version, because it uses power and time.

How do I update the model later?

You flash a new program with the new model array. Some products support updates over a wireless link. Keep a test set so you can prove the new model is really better before you update devices in the field.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Deploying and testing on a board

Deploying and testing on a board

The deployment flow

Train on your computer

Convert and quantise

Check on the computer

Turn it into C

Write the board program

Flash and test on real data

The model as a C array

A sketch of the board program

Testing on the real board

Power and privacy benefits

Watch and read more

The problem

You are done when

Hard questions

Questions people ask