Small and on-device modelsAdvancedLesson 2336 min read

Serving small models cheaply

Serving a model to many users costs money every hour. Learn batching, caching, CPU inference and autoscaling, and how to tell when a hosted API is simply cheaper.

✦ Ask the tutor🧪 Lab · 35 min📝 Lesson test
Lesson in motion

In 60 seconds

Serving small models cheaply

Serving a model to many users costs money every hour. Learn batching, caching, CPU inference and autoscaling, and how to tell when a hosted API is simply cheaper.

1/5
In simple words
A tea stall does not boil one cup at a time. It boils a big pot and pours many cups together. When few people come, it makes a smaller pot. Serving AI cheaply works the same way.
Running a model for one person on a laptop is easy. Serving it to many users is a different problem: you pay for the server every hour, whether it is busy or idle. The goal is to do as much useful work as possible for each rupee of hardware, and to compare that fairly with paying a hosted API per token.

Four ways to cut cost

TechniqueWhat it doesWhen it helps
BatchingProcesses several requests together, so the hardware is kept busyMany users at once; continuous batching is covered in Step 17
CachingStores answers or work that repeat, so you do not compute them againCommon questions; the same long system prompt on every request
CPU inferenceRuns a small quantised model on ordinary CPU serversLow or steady traffic, small models, tasks where a slower reply is fine
AutoscalingAdds servers when busy and removes them when quiet, sometimes down to zeroTraffic that goes up and down, like exam season or festival sales

Caching, two kinds

  • Response cache: if the same question comes again (for example "What are the library hours?"), return the stored answer. Only do this for answers that do not depend on the user and do not change often.
  • Prefix or prompt cache: many runtimes can reuse the work done on a long, repeated start of the prompt, such as the system prompt. This saves time and compute on each request. Check what your runtime supports.
Watch out
Never let a response cache return one user's private answer to another user. Include the user or permission level in the cache key, or do not cache personal answers at all.

Self-hosted or hosted API? Do the maths

A hosted API charges per token, so cost grows with use and is near zero when idle. A self-hosted server costs about the same every hour whether busy or not, plus your team's time. So low traffic usually favours an API, and steady high traffic can favour self-hosting.
Python: break-even between a hosted API and a self-hosted server (made-up numbers)python
# All prices below are MADE-UP examples for practice. Use real quotes for real decisions.
api_price_per_million_tokens = 50.0      # rupees, input and output combined, example only
server_cost_per_month = 20_000.0         # rupees, including storage and bandwidth, example only
ops_hours_per_month = 10                 # your team's time to run and update it
ops_cost_per_hour = 800.0                # rupees, example only

tokens_per_request = 1_500
self_hosted_monthly = server_cost_per_month + ops_hours_per_month * ops_cost_per_hour

for requests_per_day in (500, 5_000, 50_000):
    monthly_tokens = requests_per_day * 30 * tokens_per_request
    api_monthly = monthly_tokens / 1_000_000 * api_price_per_million_tokens
    cheaper = "API" if api_monthly < self_hosted_monthly else "self-hosted"
    print(f"{requests_per_day:>6} req/day: API Rs {api_monthly:,.0f} vs self-hosted Rs {self_hosted_monthly:,.0f} -> {cheaper}")
The real comparison has more parts: can one server handle your peak traffic, what quality does each option give on your test set, and what are the privacy and data-location needs? But this simple sum already stops many bad decisions.

A cheap serving checklist

  1. 1

    Measure your traffic

    Requests per day, peak per minute, average tokens in and out.
  2. 2

    Pick the smallest model that passes

    Use your test set from earlier lessons.
  3. 3

    Quantise and cache

    Lower-bit weights, prompt caching, and a response cache for safe repeated questions.
  4. 4

    Set limits

    Maximum input length, maximum output length, and per-user rate limits.
  5. 5

    Scale with demand

    Autoscale, and consider scale to zero if slow first requests are acceptable.
  6. 6

    Re-check every few months

    Prices, models and traffic all change.
Wasteful
  • A big GPU server running all night for ten users
  • No limit on answer length
  • Choosing self-hosting "because APIs are expensive" without doing the maths
Cheap and sensible
  • A hosted API while traffic is small, self-hosting when the numbers say so
  • Short, capped answers and cached repeated questions
  • A written cost comparison, updated when prices change
Note
Scale to zero saves money but the first request after a quiet period must wait for the model to load (a "cold start"). For a chatbot that users expect to answer instantly, keep at least one small server warm during busy hours.
Try this. Change the made-up numbers in the code to match a project you know. At how many requests per day does self-hosting become cheaper?

Watch and read more

llama.cpp server (llama-server)ggml-org · reading

Lab

A cost comparison between a hosted API and a self-hosted small model for one realistic app.

~35 min

The problem

App: "a question-answering bot for a coaching institute, used by students mostly in the evenings and heavily during exam season."

  1. Estimate traffic for a normal month and an exam month: requests per day, average tokens per request, and peak requests per minute.
  2. Find current public prices for one hosted API option and one server option (a cloud CPU or GPU server). Write the date you checked. If you cannot find them, use clearly labelled made-up numbers.
  3. Adapt the break-even script from the lesson. Add your team's time. Compute both months.
  4. Write down which techniques you would use: batching, caching, CPU inference, autoscaling, limits. Say why for each.
  5. Make a recommendation in three sentences, including the traffic level at which you would switch.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Self-hosting looks cheaper in exam month but more expensive in normal months. What designs could get the best of both?Reveal

Use a hosted API for normal traffic and add self-hosted capacity only for the peak, or use autoscaling so self-hosted servers run only when busy. Another option is a hybrid: answer common questions from a cache or a small self-hosted model, and send hard questions to the hosted API. Each adds complexity, so check the savings are worth it.

Q2Why can doubling the batch size make each request cheaper but also make some users wait longer?Reveal

Bigger batches keep the hardware busier, so the cost per request falls. But the server may wait to collect more requests, and each step handles more work, so individual requests can take longer to start or finish. Continuous batching reduces this problem, but there is still a trade-off between throughput (cost) and latency (waiting time) that you set with limits and testing.

Please sign in to continue.

Questions people ask

Is CPU serving good enough for real users?

For small quantised models, short answers and modest traffic, it can be. For long answers or many users at once, a GPU is usually more cost-effective per request. Measure speed on the actual server before deciding.

What does "scale to zero" mean?

When there are no requests, all model servers are shut down, so you pay almost nothing. The next request has to wait while a server starts and loads the model. It suits internal tools and background jobs more than live chat.

Why include my team's time in the cost?

Someone has to update the server, patch security issues, watch for failures and upgrade the model. That time has a real cost, and it is often the part people forget when comparing with an API.

Are hosted API prices stable?

No. Prices change, often downwards, and new cheaper models appear. Keep your comparison in a small script or spreadsheet so you can re-run it with current prices.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress