Build an LLMAdvancedLesson 476 min read

Step 11 · Modern architecture upgrades

What separates a 2019 transformer from a 2026 one. Six changes, each small, together substantial.

🧪 Lab · 20 min 📝 Lesson test

Lesson in motion

In 60 seconds

Step 11 · Modern architecture upgrades

What separates a 2019 transformer from a 2026 one. Six changes, each small, together substantial.

1/4

In simple words

Same car, better parts. Nothing about the engine changed shape — the pistons just got lighter and the gearbox got smarter.

Change	Replaces	Why
RoPE	Learned position embeddings	Relative distance, better extrapolation
RMSNorm	LayerNorm	Cheaper, no measured loss
SwiGLU	ReLU/GELU MLP	Better quality per parameter
No biases	Bias terms everywhere	Fewer parameters, more stable
GQA	Multi-head attention	Much smaller KV cache at inference
MoE	Dense feed-forward	More parameters at the same compute per token

Grouped-query attention: the inference win

At generation time you cache the key and value vectors for every previous token — the KV cache. With standard multi-head attention that cache is enormous and dominates memory during serving.

GQA lets several query heads share one key-value head. Thirty-two query heads with eight KV heads cuts the cache by four with almost no quality loss.

Grouped-query attentionpython

class GQA(nn.Module):
    def __init__(self, d_model, n_head, n_kv_head):
        super().__init__()
        self.n_head, self.n_kv = n_head, n_kv_head
        self.d_head = d_model // n_head
        self.rep = n_head // n_kv_head          # query heads per kv head
        self.q  = nn.Linear(d_model, n_head * self.d_head, bias=False)
        self.kv = nn.Linear(d_model, 2 * n_kv_head * self.d_head, bias=False)
        self.proj = nn.Linear(d_model, d_model, bias=False)

    def forward(self, x):
        B, T, C = x.shape
        q = self.q(x).view(B, T, self.n_head, self.d_head).transpose(1, 2)
        k, v = self.kv(x).view(B, T, 2, self.n_kv, self.d_head).unbind(2)
        k = k.transpose(1, 2).repeat_interleave(self.rep, dim=1)
        v = v.transpose(1, 2).repeat_interleave(self.rep, dim=1)
        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        return self.proj(y.transpose(1, 2).reshape(B, T, C))

Mixture of experts: more brain, same bill

Replace one feed-forward network with many, and route each token to only a couple of them. The model holds far more parameters, but each token only activates a small fraction, so compute per token stays roughly flat.

Total parameters and active parameters are different numbers. A sparse model can hold hundreds of billions of parameters while costing what a much smaller dense model costs per token.

What MoE gives you

Far more knowledge capacity for the same compute per token.
Faster training to a given loss.
Experts specialise, sometimes interpretably.

What MoE costs you

All parameters must be in memory, even inactive ones — serving needs a lot of VRAM.
Load balancing is fiddly; without an auxiliary loss a few experts get everything.
Trickier to distribute, fine-tune and quantise.
Higher variance in latency.

Watch out

Do not add MoE to a first project. It multiplies the number of ways your run can fail while the gain only shows up at scale. Get a dense model working end to end first.

A sensible 2026 default configuration

If you are starting freshpython

config = dict(
    norm="rmsnorm", norm_position="pre",
    position="rope", rope_theta=500000,     # large theta for long context
    ffn="swiglu", ffn_mult=8/3,
    attention="gqa", n_head=32, n_kv_head=8,
    bias=False,
    tie_embeddings=True,                    # for models under ~2B
    vocab_size=128256,
    dtype="bfloat16",
)

Watch and read more

Mixture of experts explainedYouTube search Llama architecture walkthroughYouTube search

Lab

GQA implemented, with the KV cache measured before and after.

~20 min

The problem

Implement grouped-query attention. Measure KV cache size in MB at 4k context for MHA vs GQA-8 vs MQA, and measure the quality difference on a held-out set.

Starter codepython

def kv_cache_mb(n_layers, n_kv_heads, d_head, seq, batch, bytes_=2):
    return 2 * n_layers * n_kv_heads * d_head * seq * batch * bytes_ / 1e6
print(kv_cache_mb(32, 32, 128, 4096, 8), "MB  MHA")
print(kv_cache_mb(32,  8, 128, 4096, 8), "MB  GQA-8")

You are done when

Cache size measured for three configurations
Quality difference measured, not assumed
You can state the concurrent-user implication

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your KV cache is 40GB at your target batch size and context. List every lever, with its cost.Reveal

GQA or MQA — 4-8x reduction, small quality cost. KV cache quantisation to int8 — 2x, small accuracy cost. Shorter context — free, product cost. Smaller batch — free, throughput cost. Sliding-window or local attention on most layers — large reduction, loses exact long-range recall. Paged attention — no reduction but removes fragmentation, often 2x effective capacity. Start with GQA and paged attention; they cost the least.

Please sign in to continue.

Questions people ask

Should I use MoE?

Only if you are training at meaningful scale and have the serving memory. For anything under about 10B active parameters, a dense model is simpler and usually the better engineering choice.

Does GQA hurt quality?

Slightly, and it is close to free in practice. The usual ratio of 4:1 or 8:1 query-to-KV heads shows very small quality loss for a large serving win. Multi-query attention — one KV head — goes further and costs a bit more quality.

What is rope_theta and why raise it?

It sets the base frequency of the rotation. A larger value spreads the angles more slowly across positions, which makes long-context extension work better. Models targeting long context commonly use 500k or higher instead of the original 10k.

Are there real alternatives to transformers?

State-space models such as Mamba are genuinely competitive at some scales with linear-time inference, and hybrid designs interleave both. Attention still dominates production, but this is the most interesting active area.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.