TinyML: AI on tiny chipsMiddleLesson 2785 min read

Shrinking it to fit

int8 quantisation stores each weight in one byte instead of four. Do it yourself in numpy, and measure how much size you save and how much accuracy you lose.

✦ Ask the tutor 🧪 Lab · 25 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Shrinking it to fit

int8 quantisation stores each weight in one byte instead of four. Do it yourself in numpy, and measure how much size you save and how much accuracy you lose.

1/5

In simple words

Imagine measuring every student's height to the exact tenth of a millimetre. That is a lot of writing, and nobody needs it. If you round to the nearest centimetre, each number is shorter, and you can still line everyone up correctly. Quantisation rounds a model's numbers so they take less space, while the model still gives the same answers most of the time.

A trained model's weights are usually float32 numbers: 4 bytes each. int8 stores each weight as a whole number from −128 to 127, in 1 byte. That makes the weights four times smaller. On many microcontrollers, int8 maths is also faster and uses less energy.

Symmetric int8 quantisation, step by step

1
Find the largest weight
Take the largest absolute value in the layer, for example 1.3.
2
Choose a scale
scale = largest ÷ 127. Now 127 in int8 means the largest weight.
3
Divide and round
q = round(weight ÷ scale), clipped to −127 … 127, stored as int8.
4
Use it
To get an approximate weight back: weight ≈ q × scale. The error is at most half a step.

Python: quantise six weights and check the errorpython

import numpy as np

w = np.array([0.83, -0.312, 0.051, -1.3, 0.644, 0.002])
scale = np.abs(w).max() / 127          # one float for the whole layer
q = np.round(w / scale).astype(np.int8)
back = q * scale

print("scale:", round(scale, 5))
print("int8: ", q)
print("back: ", back.round(4))
print("error:", np.abs(w - back).round(4))

Each weight moved by at most half of scale. Small weights lose the most, as a share of their size. The last weight, 0.002, became 0: it was smaller than half a step.

Real TinyML quantisation does a bit more

Activations too. On the chip, the inputs and the in-between values are also int8, so the whole model runs in integer maths. The converter needs a small set of real example inputs, often called a representative dataset, to choose scales for them.
Zero points. Values that are not centred on zero (for example, all positive) use an extra offset called a zero point, so the int8 range is not wasted.
Per-channel scales. Each output channel of a layer can get its own scale, which usually loses less accuracy than one scale for the whole layer.
Biases are often kept at higher precision (for example int32), because they are few and matter a lot.
Quantisation-aware training lets the model practise with rounding during training, which can recover accuracy when simple rounding after training loses too much.

Note

The earlier lesson Quantisation in practice covers quantising large models on phones and laptops. The idea is the same. On a microcontroller, full int8 is usually a must, not an option, because of memory and because many chips are slow at floating point.

Careless shrinking

Quantise and ship without measuring
Check only overall accuracy
Use a few random inputs as the representative dataset
Assume Python and board give the same answers

Careful shrinking

Measure size and held-out accuracy before and after
Check each class, especially rare ones
Use real inputs that cover the conditions the device will see
Compare board outputs with Python outputs on the same inputs

Watch out

If accuracy drops a lot after quantisation, do not just accept it. Check for a layer with a few very large weights (one outlier makes the scale big and every other weight coarse), try per-channel scales, or try quantisation-aware training. Check your tool's current docs for the options it supports.

Try this. In the code above, change 1.3 to 13.0. What happens to the other weights after rounding? This is why one outlier can hurt a whole layer.

Watch and read more

Lab

Your own int8 quantise and dequantise functions, with the size saved and the accuracy change measured.

~25 min

The problem

The starter trains a small neural network on scikit-learn's built-in 8×8 handwritten digits: a stand-in for a tiny camera reading a meter display.

Write quantise(w): scale = largest absolute weight ÷ 127; q = round(w ÷ scale), clipped to −127 … 127, as np.int8. Return (q, scale).
Write dequantise(q, scale): return q × scale as floats.
Run it. Write down the weight size in bytes before and after, and the accuracy before and after.
Only the weights are quantised here; the biases stay float. Write one sentence on why a real board would also quantise the activations.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import copy
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

# A tiny image model: 8x8 pictures of handwritten digits (like reading a meter display).
X, y = load_digits(return_X_y=True)
X = X / 16.0
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
model = MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000, random_state=0).fit(X_train, y_train)

def quantise(w):
    """Symmetric int8: scale = largest |weight| / 127, q = round(w / scale).
    Return (q as an np.int8 array, scale)."""
    raise NotImplementedError("Find the scale, divide, round, clip to -127..127, convert to np.int8")

def dequantise(q, scale):
    """Turn int8 numbers back into approximate float weights."""
    raise NotImplementedError("Multiply by the scale")

float_bytes, int8_bytes = 0, 0
small = copy.deepcopy(model)
for i, w in enumerate(model.coefs_):
    q, scale = quantise(w)
    small.coefs_[i] = dequantise(q, scale)
    float_bytes += w.astype(np.float32).nbytes
    int8_bytes += np.asarray(q).nbytes
    print(f"layer {i}: {w.shape}, largest error {np.abs(w - small.coefs_[i]).max():.4f}")

print(f"weights: float32 {float_bytes} bytes -> int8 {int8_bytes} bytes")
print("float accuracy:", round(model.score(X_test, y_test), 3))
print("int8  accuracy:", round(small.score(X_test, y_test), 3))

Output

No output. Add print(...) to see values.

You are done when

quantise returns an int8 array using the full −127 … 127 range, and a scale
Every weight comes back within half a step of its original value
The weight size is a quarter of the float32 size
Accuracy is measured before and after, and the change is small

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why do we round instead of just cutting off the decimals?Reveal

Rounding picks the nearest int8 value, so the error is at most half a step. Cutting off (truncating) always moves values towards zero, so the error can be almost a whole step and is biased in one direction. That bias adds up across many weights and hurts accuracy more.

Q2Your quantised model is accurate in Python, but the board's answers are different. What could cause this?Reveal

The board may quantise the input differently: a different scale or zero point, or the input not converted to int8 in the same way. The runtime also rounds activations between layers, which a weights-only test in Python does not show. Feed the same input to both, and compare the input tensor and outputs step by step. The converter's own tools can often run the quantised model on your computer for this comparison.

Please sign in to continue.

Questions people ask

Does quantisation always lose accuracy?

Almost always a little, sometimes none you can measure, and sometimes a lot. It depends on the model and the data. That is why you measure on held-out data before and after.

Why −127 to 127 and not −128 to 127?

Symmetric schemes often use −127 to 127 so that positive and negative values have the same range and zero is exactly zero. Some tools use −128 for special cases. Either way, follow what your runtime expects.

Can I go smaller than int8?

Some research and some tools use 4-bit or even fewer bits per weight, and some chips support special formats. It saves more space but loses more accuracy and has less support on microcontrollers. int8 is the common, well-supported choice today; check your tool's docs.

What else shrinks a model besides quantisation?

Using fewer or smaller layers, fewer input features, pruning (removing weights that matter little) and distillation (training a small model to copy a larger one). For TinyML, starting with a small design is usually the biggest win.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Shrinking it to fit

Shrinking it to fit

Symmetric int8 quantisation, step by step

Find the largest weight

Choose a scale

Divide and round

Use it

Real TinyML quantisation does a bit more

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask