Build an LLMMiddleLesson 376 min read

Step 1 · The tokenizer

Before a model sees language, something must chop text into pieces. Get this wrong and everything downstream is quietly worse.

🧪 Lab · 25 min📝 Lesson test
Lesson in motion

In 60 seconds

Step 1 · The tokenizer

Before a model sees language, something must chop text into pieces. Get this wrong and everything downstream is quietly worse.

1/5
In simple words
Computers cannot read letters. So we make a dictionary: every common chunk of letters gets a number. "Hello" might be number 15496. The model only ever sees numbers.
A tokenizer maps text to a list of integers and back. Three choices, and modern models all pick the third:
ApproachVocabularyProblem
One token per character~100Sequences become enormously long; the model wastes capacity on spelling
One token per wordMillions, and still incompleteEvery typo and rare name is an unknown token
Subword (BPE)30k–200kNone serious — this is why everyone uses it

Byte-pair encoding, in one paragraph

Start with every byte as its own token. Count which pair of adjacent tokens occurs most often in your corpus. Merge that pair into one new token. Repeat a few thousand times. Common words end up as single tokens; rare words break into recognisable pieces; nothing is ever unknown, because you can always fall back to raw bytes.
Train a byte-pair encoderpython
from collections import Counter

def get_pairs(ids):
    return Counter(zip(ids, ids[1:]))

def merge(ids, pair, new_id):
    out, i = [], 0
    while i < len(ids):
        if i < len(ids) - 1 and (ids[i], ids[i+1]) == pair:
            out.append(new_id); i += 2
        else:
            out.append(ids[i]); i += 1
    return out

def train_bpe(text, vocab_size=512):
    ids = list(text.encode("utf-8"))       # start: raw bytes, 0-255
    merges = {}
    for new_id in range(256, vocab_size):
        pairs = get_pairs(ids)
        if not pairs: break
        best = max(pairs, key=pairs.get)   # most frequent adjacent pair
        ids = merge(ids, best, new_id)
        merges[best] = new_id
    return merges

def encode(text, merges):
    ids = list(text.encode("utf-8"))
    for pair, new_id in merges.items():    # apply in training order
        ids = merge(ids, pair, new_id)
    return ids

def decode(ids, merges):
    vocab = {i: bytes([i]) for i in range(256)}
    for (a, b), new_id in merges.items():
        vocab[new_id] = vocab[a] + vocab[b]
    return b"".join(vocab[i] for i in ids).decode("utf-8", errors="replace")
That is a working tokenizer in forty lines. Production versions add a regex pre-split (so tokens never straddle a word boundary in a silly way), special tokens, and a fast Rust implementation — but the algorithm above is the real one.

Why tokenizer decisions haunt you

  1. 1

    Vocabulary size is a trade

    Bigger vocabulary means shorter sequences and faster inference, but a larger embedding matrix and more rarely-seen tokens. 32k to 128k is the normal range today.
  2. 2

    Language coverage is political and practical

    If your corpus was mostly English, Hindi or Tamil text costs three to five times as many tokens to say the same thing. That is a real price and latency penalty for those users, baked in at tokenizer time.
  3. 3

    Numbers and code need care

    Splitting "12345" into odd chunks damages arithmetic. Most modern tokenizers split digits individually on purpose.
  4. 4

    You cannot change it later

    The embedding table is indexed by token id. Change the tokenizer and every weight you trained is meaningless. Decide once.
Watch out
This is also why models struggle to count letters in a word or reverse a string. They never see letters — they see chunks. Asking "how many r's in strawberry" is asking someone to count things they cannot see.
Do this
Practical advice: unless you are pretraining at T3 or above, do not train your own tokenizer. Reuse the one from the model you are building on. Mismatched tokenizers are one of the most common silent bugs in fine-tuning.

Watch and read more

Let's build the GPT TokenizerAndrej Karpathy · 2 hr · video
Tokenization deep diveYouTube search

Lab

A BPE tokenizer you wrote, benchmarked against a real one.

~25 min

The problem

Implement BPE training and encoding from Module 37. Train it on a few MB of text. Then measure tokens-per-word for English and for an Indian language, and compare against a production tokenizer.
Starter codepython
# Measure the fertility gap that decides who pays more per request
for lang, text in samples.items():
    n_tokens = len(encode(text, merges))
    n_words = len(text.split())
    print(f"{lang:10} {n_tokens/n_words:.2f} tokens/word")

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your tokenizer needs 3.5 tokens per Hindi word and 1.3 per English word. State every consequence.Reveal
Hindi users pay ~2.7x more per request and wait longer; their effective context window is ~2.7x smaller in words, so long documents truncate sooner; more of the model's capacity is spent on tokenisation rather than meaning, which measurably lowers quality; and at fixed training compute, Hindi text contributes fewer words of learning per token spent. Tokenizer fairness is a product decision that looks like an implementation detail, and it is baked in permanently at training time.

Please sign in to continue.

Questions people ask

How many tokens is a word?

English averages roughly 1.3 tokens per word, or about 4 characters per token. Code, non-Latin scripts and unusual names cost more. Always measure on your actual data rather than trusting the rule of thumb.

What are special tokens?

Reserved ids the text can never produce naturally: end-of-text, padding, and chat-role markers like beginning-of-turn. They give the model unambiguous structure. Get them wrong in fine-tuning and the model will not know when to stop generating.

Why do some models use SentencePiece?

SentencePiece treats the input as a raw stream including whitespace, which avoids language-specific pre-tokenisation rules. It suits multilingual models. Byte-level BPE and SentencePiece unigram are the two dominant families.

Can I extend a vocabulary?

Yes — add new tokens and grow the embedding matrix, initialising new rows sensibly (often the mean of the sub-token embeddings). Useful for adding a domain vocabulary or a new script. It requires further training to be worth anything.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress