Fine-tuning a small model for one job
Fine-tuning teaches a small model one narrow task. Learn when it is worth it, how to prepare data, how LoRA and QLoRA fit on one GPU, and how to measure before and after.
In 60 seconds
Fine-tuning a small model for one job
Fine-tuning teaches a small model one narrow task. Learn when it is worth it, how to prepare data, how LoRA and QLoRA fit on one GPU, and how to measure before and after.
Try these first
- 1
Better prompts
Clear instructions and a few examples in the prompt. Cheapest and fastest to change. - 2
Retrieval (RAG)
If the model lacks facts, give it the right documents. Fine-tuning is poor at adding lots of new facts. - 3
A different model
Another small model may already do the task well. - 4
Then fine-tune
When you need a steady format, style or skill that prompts cannot get reliably.
| Good reasons to fine-tune | Weak reasons |
|---|---|
| Output must follow a strict format every time | You want the model to learn your whole company wiki |
| A narrow classification or extraction task with many examples | You have only ten examples |
| A small model must replace a bigger, costlier one for one job | Prompting has not really been tried yet |
| A special tone or domain style | The task changes every week |
Preparing data
- Format: many tools accept JSON Lines, one example per line, often as chat messages with a user turn and the ideal assistant turn.
- Quality over quantity: a few hundred clean, varied, correct examples often beat many thousands of noisy ones.
- Cover the real mix: include short and long inputs, typing mistakes, Hinglish and other mixed-language text if your users write that way.
- Split first: put aside a test set before training, and never train on it.
- Privacy: remove names, phone numbers, Aadhaar-like numbers and other personal data unless you have a clear lawful reason and consent.
import json, random
def load(path):
with open(path, encoding="utf-8") as f:
return [json.loads(line) for line in f if line.strip()]
rows = load("examples.jsonl") # each row: {"input": "...", "output": "..."}
seen, clean = set(), []
for r in rows:
key = r["input"].strip().lower()
if key and r.get("output") and key not in seen:
seen.add(key)
clean.append(r)
random.seed(42)
random.shuffle(clean)
n_test = max(20, len(clean) // 10)
test, train = clean[:n_test], clean[n_test:]
for name, part in (("train.jsonl", train), ("test.jsonl", test)):
with open(name, "w", encoding="utf-8") as f:
for r in part:
f.write(json.dumps(r, ensure_ascii=False) + "\n")
print(f"{len(rows)} rows -> {len(clean)} unique -> {len(train)} train, {len(test)} test")LoRA and QLoRA on a small budget
transformers, peft and trl libraries, and bitsandbytes for 4-bit loading. Their APIs change between versions, so follow their current documentation rather than old blog posts.Measure before and after
- 1
Baseline
Run the base model (with your best prompt) on the test set. Record the score. - 2
Train
Fine-tune on the training set only. Keep notes of settings and data version. - 3
Re-test
Run the fine-tuned model on the same test set, with the same scoring. - 4
Check side effects
Try a few general questions and safety prompts. Did anything get worse? - 5
Decide
Keep it only if it clearly beats the baseline on the task and did not break important things.
Watch and read more
Lab
A clean, split dataset for one narrow task and a before-and-after evaluation plan.
The problem
Task: "extract the item, quantity and delivery date from a kirana shop's WhatsApp order message, as JSON."
- Write at least 40 realistic, made-up order messages in the ways people really type (English, Hinglish, short forms, spelling mistakes), each with the correct JSON output.
- Save them as JSONL and run the cleaning and splitting script from the lesson.
- Write a scoring rule: for example, 1 point for each field exactly right.
- Run a small local model with a good prompt on the test set and record the baseline score, or on paper, write how you would do it.
- Write your fine-tuning plan: base model (with licence checked), LoRA or QLoRA, where you would train, and what score would make it worth keeping.
Practice lab
Edit the code and run it. Python runs in your browser — nothing is sent to our server.
No output. Add print(...) to see values.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1You tuned your prompt many times while looking at the test set, then fine-tuned. Why might your comparison be unfair, and how do you fix it?Reveal
By tuning the prompt on the test set, you have partly fitted it to those examples, so the baseline score may be too high or too low compared with fresh data. Keep three parts: a training set, a development set for trying prompts and settings, and a test set you only use at the end for the final comparison.
Q2Why can a fine-tuned small model be cheaper than a big model with a long prompt, even when both give correct answers?Reveal
A big model with a long prompt full of instructions and examples pays for all those input tokens on every request, and each token costs more on a bigger model. A fine-tuned small model has learned the format, so it needs a short prompt, and each token is cheaper to compute. At high traffic the difference can be large. You still have to count the cost of training and running it.
Questions people ask
Can I fine-tune without a GPU?
Training on a CPU is possible for very small models but usually very slow. Most people use a single GPU, a free notebook tier, or a short paid rental. Planning, data preparation and evaluation can all be done on any laptop.
Will fine-tuning teach the model new facts?
A little, but not reliably. It is much better at teaching format, style and a narrow skill. For facts that change, such as prices or rules, use retrieval instead.
What do I get at the end of LoRA training?
A small adapter file. You load it together with the original base model. Some tools can also merge it into the base model and then quantise the result, for example to GGUF for local use.
My fine-tuned model got worse at general chat. Is that normal?
It can happen, especially with narrow data or too much training. If general ability matters, mix in some general examples, train for less time, and always check side effects in your evaluation.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.