Distillation: teaching a small model
A big teacher model can help train a small student model for one job. Learn how knowledge distillation and synthetic data work, and the licence rules that can forbid it.
In 60 seconds
Distillation: teaching a small model
A big teacher model can help train a small student model for one job. Learn how knowledge distillation and synthetic data work, and the licence rules that can forbid it.
Two common styles
| Style | What the student learns from | Needs |
|---|---|---|
| Classic (soft labels) | The teacher's full probability for every possible answer, not only the top one | Access to the teacher's output probabilities, usually an open-weight teacher you run yourself |
| Synthetic data (text only) | Example answers the teacher writes, used as normal training data | Only the teacher's text outputs, for example through an API, if the terms allow it |
A synthetic-data pipeline
- 1
Define one job
For example: classify a Hindi or English customer message into five categories and give a short reason. - 2
Collect real inputs
Real messages, used with consent and with personal data removed, or realistic made-up ones. - 3
Ask the teacher
Have the big model produce the answer for each input, in a fixed format. - 4
Filter and check
Drop answers in the wrong format. Have people check a sample. Remove duplicates. - 5
Train the student
Fine-tune the small model on the checked examples (see the next lesson). - 6
Evaluate on held-out data
Test on examples the teacher never saw, graded by people or clear rules.
import json
LABELS = ["order", "complaint", "delivery", "refund", "other"]
def call_model(prompt: str) -> str:
... # plug in your provider or local teacher model here
def teacher_label(message: str) -> dict | None:
prompt = (
"Classify this customer message into one of: " + ", ".join(LABELS) + ".\n"
'Reply only with JSON like {"label": "...", "reason": "..."}.\n\n'
"Message: " + message
)
try:
out = json.loads(call_model(prompt))
except (json.JSONDecodeError, TypeError):
return None # drop badly formatted answers
if out.get("label") not in LABELS:
return None # drop invalid labels
return {"input": message, "label": out["label"], "reason": out.get("reason", "")}
def build_dataset(messages: list[str], path: str) -> None:
with open(path, "w", encoding="utf-8") as f:
for m in messages:
row = teacher_label(m)
if row:
f.write(json.dumps(row, ensure_ascii=False) + "\n")Check the licence and terms first
What goes wrong
- Copied mistakes: the student learns the teacher's errors and biases too. Check samples by hand.
- Narrow data: if all examples look alike, the student fails on real messages that look different. Use real, varied inputs.
- Leaked test data: if your test examples were also in training, your scores will look too good.
- Personal data: do not send people's private messages to an outside teacher without consent and a clear reason. Remove names and numbers first.
Watch and read more
Lab
A written distillation plan for one narrow task, with a licence check and an evaluation design.
The problem
Task: "a small model that reads a farmer's question (Hindi, Marathi or English) and tags it as one of: crop disease, weather, market price, government scheme, other."
- Choose a teacher model. Read its licence and terms and write down exactly what they say about using outputs to train other models. If it is not allowed, choose another teacher.
- Write the teacher prompt and the exact output format.
- Describe where your inputs come from and how you handle consent and personal data.
- Write three filtering rules for bad teacher answers.
- Design the test: how many held-out examples, who labels them, and what score the student must reach to replace the teacher.
You do not need to run anything. If you want to try the code, use the sketch in the lesson with a local model.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why might a student trained only on the teacher's most confident answers perform worse on real traffic?Reveal
Confident answers are usually the easy cases. If you train only on those, the student never sees the hard, unclear messages that real users send, so it has not learned how to handle them. Keeping some harder examples, checked by people, gives the student a more realistic picture.
Q2Your distilled student scores higher than the teacher on your test set. What should you check before celebrating?Reveal
Check that no test examples (or close copies) were in the training data. Check that the test labels came from people, not from the teacher, or you are only measuring agreement with the teacher. Look at the errors of both models by hand, and check a sample of fresh real traffic. If it still holds, it can be real for a narrow task.
Questions people ask
Can a student become better than its teacher?
On the whole, usually not. On one narrow task, with careful filtering and some human-checked data, a student can sometimes match or beat the teacher on that task. Measure it on held-out examples; do not assume it.
How many examples do I need?
There is no fixed number. For a simple classification task, a few hundred to a few thousand good, varied examples is a common starting range. Start small, measure, and add more where the student makes mistakes.
Is distillation the same as fine-tuning?
Distillation is about where the training signal comes from: a teacher model. Fine-tuning is the training step itself. In the synthetic-data style you usually fine-tune the student on the teacher's examples, so the two go together.
Can I use an open-weight model as the teacher?
Yes, and it gives you more control, including access to output probabilities. Its licence still applies, so check what it says about using outputs to train other models.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.