Which LLM can you actually build?
There are five honest tiers, separated by three or four orders of magnitude in cost. Pick yours before you write a line of code.
In 60 seconds
Which LLM can you actually build?
There are five honest tiers, separated by three or four orders of magnitude in cost. Pick yours before you write a line of code.
Interactive · pick your tier
What each tier actually teaches you
| Tier | You learn | You do not learn |
|---|---|---|
| T0 · Prompt + RAG | Context design, retrieval, evaluation, product sense | Anything about how the model works inside |
| T1 · LoRA fine-tune | Data formatting, training loops, overfitting, eval discipline | Pretraining dynamics, distributed systems |
| T2 · Full fine-tune / continued pretraining | Optimiser behaviour, memory management, multi-GPU basics | Frontier-scale data engineering |
| T3 · Pretrain a small model | Everything. Tokenizers, architecture, data, scaling, infrastructure | What breaks only at 1000+ GPUs |
| T4 · Frontier pretrain | Cluster reliability, data at trillions of tokens, months-long runs | Nothing — this is the deep end |
The route this track takes
- 1
Modules 37–42 · build one from nothing
Tokenizer, embeddings, attention, transformer block, a complete tiny GPT, and a training loop that converges. Real code you can run on one GPU or a laptop. - 2
Modules 43–48 · make it real
Data pipelines, scaling laws and compute budgets, distributed training, speed, modern architecture, long context. - 3
Modules 49–52 · make it useful
Instruction tuning, preference optimisation, reasoning training, and LoRA — the path most people should take. - 4
Module 53 · ship it
Evaluation, quantisation, serving, cost per token.
What you need before starting
- Python and a little PyTorch. If you can write a for-loop and a class, you are fine.
- Enough linear algebra to be comfortable with "a matrix multiply mixes information". No proofs required.
- A GPU is helpful, not required, for Modules 37–42. Free notebook GPUs are enough for a tiny model.
- Patience with silence. Training gives almost no feedback for long stretches. That is normal, and it is why Module 42 is mostly about instrumentation.
Watch and read more
Lab
An honest costing of the model you actually want.
The problem
N, D = 7e9, 15e12
C = 6 * N * D
gpu_hours = C / (400e12 * 0.40) / 3600
print(f"{C:.2e} FLOPs, {gpu_hours:,.0f} GPU-hours, ${gpu_hours*2:,.0f} at $2/hr")
print(f"inference: {2*N/1e9:.1f} GFLOPs per token")You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your calculation says training costs $200k and API access costs $8k/year for the same quality. Name three reasons to train anyway.Reveal
Questions people ask
Do I need a PhD?
No. The core transformer is about 200 lines of readable code. What separates practitioners is data judgement and evaluation discipline, both of which are learned by doing.
Should I use PyTorch or JAX?
PyTorch, unless you are joining a team that uses JAX. More tutorials, more code to read, more jobs. The concepts transfer completely.
Can I train something useful on one GPU?
A LoRA fine-tune of a 7B model, absolutely — that is a genuinely useful, shippable product on a single 24 GB card. Pretraining from scratch on one GPU tops out around 100M–500M parameters on a small dataset, which is educational rather than useful.
Is it cheaper to fine-tune or to prompt well?
Prompt well first, every time. Fine-tuning is worth it when you need a consistent format, a narrow domain, lower latency, or a smaller model doing a job a big one currently does. It is rarely the fix for "the model does not know my facts" — that is retrieval.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.
Sign in with your phone number to take the test and save your progress