Small and on-device modelsMiddleLesson 2287 min read

Running a model on your own computer

Tools like llama.cpp and Ollama let you run open-weight models on a laptop. Learn how much memory you need, when a GPU helps, and what a GGUF file is.

✦ Ask the tutor 🧪 Lab · 40 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Running a model on your own computer

Tools like llama.cpp and Ollama let you run open-weight models on a laptop. Learn how much memory you need, when a GPU helps, and what a GGUF file is.

1/3

In simple words

A recipe book is useless without a kitchen. The model file is the recipe book. A program like llama.cpp or Ollama is the kitchen that reads it and cooks the answer, right in your own house.

To run a model locally you need two things: the model file (the weights) and a runtime, a program that loads the weights and generates text. Two well-known free runtimes are llama.cpp, an open-source C/C++ project, and Ollama, a tool that makes downloading and running models simple and is built on llama.cpp's work.

GGUF: the file format

llama.cpp and Ollama commonly use GGUF files. A GGUF file packs the weights, the tokenizer and settings into one file, often already quantised (made smaller, see the next lesson). You will see names with tags like Q4 or Q8. These roughly tell you how many bits each weight uses.

How much memory do you need?

A good first guess: number of parameters × bytes per parameter, then add some extra for the runtime and the conversation (the context).

Precision	Bytes per parameter	1B-parameter model	7B-parameter model
16-bit	2	about 2 GB	about 14 GB
8-bit	1	about 1 GB	about 7 GB
4-bit	0.5	about 0.5 GB	about 3.5 GB

These are the weights only. Long conversations need more memory, because the model keeps a cache of the text so far (the KV cache). Leave room for your operating system and browser too. On an 8 GB laptop, a small 4-bit model is a comfortable place to start.

Python: rough memory estimate (standard library only)python

def estimate_gb(params_billion: float, bits: int, overhead: float = 1.2) -> float:
    """Weights only, times a rough overhead factor for runtime and context."""
    bytes_per_param = bits / 8
    weights_gb = params_billion * bytes_per_param   # 1B params x 1 byte = about 1 GB
    return round(weights_gb * overhead, 1)

for size in (1, 3, 7):
    for bits in (16, 8, 4):
        print(f"{size}B model at {bits}-bit: about {estimate_gb(size, bits)} GB")

CPU or GPU?

Hardware	Good for	Watch out for
CPU only	Small models, learning, low traffic. Works on almost any laptop.	Slower generation, especially for bigger models
Laptop or desktop GPU	Much faster generation if the model fits in the GPU's own memory (VRAM)	If the model is bigger than VRAM, part of it runs on the CPU and speed drops
Apple silicon Mac	CPU and GPU share one memory pool, which llama.cpp can use	Shared memory is also used by everything else you run

Getting started (examples: check the current docs)

Commands change between versions, so treat these as examples and check the official pages for Ollama and llama.cpp.

Example commands (may change; check current docs)bash

# Ollama: download a model and chat with it in the terminal
ollama run <model-name>
ollama list                 # models you have downloaded

# llama.cpp: run a GGUF file you downloaded yourself
llama-cli -m ./my-model.gguf -p "Explain UPI in two sentences."

# llama.cpp: start a local server your own code can call
llama-server -m ./my-model.gguf --port 8080

Ollama also runs a local HTTP server, by default on port 11434, so your own programs can call the model. Nothing needs to leave your computer.

Python: call a local Ollama model (requests package; check the API docs)python

import requests

resp = requests.post(
    "http://localhost:11434/api/generate",
    json={"model": "<model-name>", "prompt": "Give three tips for saving water at home.", "stream": False},
    timeout=120,
)
resp.raise_for_status()
print(resp.json()["response"])

Watch out

Only download model files from sources you trust, such as the publisher's official page. Some older model file formats, based on Python's pickle, can run code when loaded. GGUF and safetensors files are designed to hold only data, but a tampered or badly made model can still give bad answers.

Note

The local server is meant for your own machine. Do not expose it to the internet without authentication and limits, or strangers may use your computer's power.

Try this. Check your laptop's RAM (and GPU memory, if any). Using the table, write the largest model size you could run at 4-bit with room to spare.

Watch and read more

llama.cpp on GitHubggml-org · reading OllamaOllama · reading

Lab

A small model running on your own computer, or a written plan for running one.

~40 min

The problem

Option A (if you can install software):

Install Ollama or build llama.cpp by following its official instructions.
Pick a small model whose licence you have read. Run it and ask it five questions: two in English, two in an Indian language you know, and one about a fact you can check.
Record: model name and size, quantisation, time to first answer, whether each answer was correct, and your RAM use (from your system monitor).

Option B (on paper):

Write down your computer's RAM, CPU and GPU (if any).
Use the memory table or the Python estimate to choose a model size and quantisation that fits with room to spare.
Write the steps you would follow, where you would download the model from, and five test questions with the answers you expect.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

Output

No output. Add print(...) to see values.

You are done when

The chosen model fits in memory with room left for the system
The model's licence was checked before use
Answers were checked for correctness, including at least one in an Indian language
Speed and memory use are recorded (Option A) or estimated with reasons (Option B)

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Two GGUF files of the same model are labelled Q8 and Q4. The Q4 one answers your questions just as well. Why might you still keep testing?Reveal

Five questions are too few to see small quality losses. The lower-bit file might fail more on harder questions, longer answers, or Indian languages, which you did not test much. Build a larger test set from real tasks, then compare the two on the same set before deciding.

Q2Why is "only use half your RAM for the model" a sensible rule on a laptop?Reveal

The operating system, browser and other apps also need memory, and the model's context cache grows during long conversations. If memory runs out, the computer starts swapping to disk and everything becomes very slow. Leaving room keeps both the model and your machine usable.

Please sign in to continue.

Questions people ask

I do not have a GPU. Can I still run a model?

Yes. llama.cpp and Ollama run on the CPU. Small models are usable for learning and light work. Generation will be slower than with a GPU, so start small.

Why is the first answer slow and later ones faster?

The first time, the runtime has to load the model file from disk into memory. After that it stays loaded for a while, so answers start sooner.

What is the difference between llama.cpp and Ollama?

llama.cpp is the lower-level engine with many options. Ollama wraps a similar engine in a simpler tool with a model library and a local API. Beginners often start with Ollama; people who want fine control often use llama.cpp directly.

My laptop gets very hot. Is that a problem?

Running a model uses the processor heavily, so heat is normal. Keep air vents clear, use a hard surface, and stop if the machine becomes unstable. Smaller models and shorter answers reduce the load.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.