Why small models matter
Big models are impressive, but small ones can be cheaper, more private, faster and able to work offline. Learn the real trade-offs so you can choose well.
In 60 seconds
Why small models matter
Big models are impressive, but small ones can be cheaper, more private, faster and able to work offline. Learn the real trade-offs so you can choose well.
Four reasons to go small
| Reason | What it means | Indian example |
|---|---|---|
| Cost | No per-token bill to a provider. You pay for your own hardware and electricity instead. | A student project that must run for months on a tiny budget |
| Privacy | Data does not leave the device or your own server. | A clinic that cannot send patient notes to an outside service |
| Offline use | Works with no internet, or with a weak and costly connection. | A farm advisory app used in a village with patchy mobile data |
| Latency | No round trip over the network, so short answers can come back quickly. | Instant autocomplete in a keyboard app |
What you give up
- Knowledge: a small model remembers fewer facts, so it may make up answers more often. Giving it the right documents (RAG) helps.
- Reasoning: long, multi-step problems are harder for small models.
- Languages: quality in Indian languages can drop more than in English. Test in the languages your users actually speak.
- Your time: you now run, update and secure the model yourself. That work is a real cost.
- Picking the biggest model for every task because it is "best"
- Picking a tiny model only because it is free, without testing
- Comparing models on public leaderboards only
- Writing down the task, the users, the budget and the privacy needs first
- Testing two or three sizes on 30 to 50 of your own examples
- Choosing the smallest model that passes your test
A simple decision path
- 1
Can the data leave the device?
If no, you need a local or self-hosted model. - 2
Must it work offline?
If yes, the model must run on the device itself. - 3
How hard is the task?
Narrow and repetitive tasks suit small models. Open-ended research and hard reasoning may not. - 4
How much traffic?
Very low traffic is often cheaper on a hosted API. Steady high traffic may be cheaper self-hosted (lesson 8). - 5
Test before you decide
Run your own examples on each candidate and compare.
Watch and read more
Lab
A one-page decision note: small local model or big hosted model, for one real task.
The problem
Pick one task you care about, for example: "sort customer WhatsApp messages for a small bakery into order, complaint and question" or "answer questions about my college's exam rules".
- Write the task in two lines: input, output, who uses it.
- Answer four questions: Can the data leave the device? Must it work offline? How hard is the task? How many requests per day?
- Make a table with two columns, Small local model and Big hosted model, and rows for cost, privacy, offline use, latency, quality and your own effort.
- Write your choice in one sentence, and one thing you would test before trusting it.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1A small model gets 90 of your 100 test examples right, and a big hosted model gets 96. When could the small model still be the better choice?Reveal
When its mistakes are cheap to catch or fix, for example because a person reviews every output anyway, and when privacy, offline use or cost matter a lot. It is also worth looking at which examples each model fails: if the small model fails only on rare cases, you could send just those cases to a person or to the big model.
Q2Why is "the smallest model that passes your test" a better rule than "the best model you can afford"?Reveal
A smaller model is usually cheaper, faster and easier to run and update, and it can fit on more devices. If it already meets your quality bar, the extra quality of a bigger model may not help users, while its extra cost and complexity stay with you. The rule also forces you to write a test first, which is the most useful step anyway.
Questions people ask
Is a small model always worse than a big one?
On broad, hard tasks, usually yes. On one narrow task that it has been tested or fine-tuned for, a small model can be good enough, and sometimes as good as a much bigger general model. Only your own test tells you.
How small is "small"?
There is no fixed number. It depends on your hardware. A useful rule: if it runs on the device or single machine you actually have, at a speed your users accept, it is small enough.
Is local AI automatically private?
The model itself does not send your data anywhere, but the app around it might, for example through logs, analytics or crash reports. Check the whole app, not only the model.
Should beginners start with small models?
They are a great way to learn, because you can see and control everything. For a real product, start with the question "what does the task need?", not with a favourite model.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.