Multimodal and generative AIMiddleLesson 2446 min read

How diffusion models make images

A diffusion model learns to remove noise, a little at a time. Start from pure static, clean it step by step with your words as a guide, and a picture appears.

✦ Ask the tutor 🧪 Lab · 30 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

How diffusion models make images

A diffusion model learns to remove noise, a little at a time. Start from pure static, clean it step by step with your words as a guide, and a picture appears.

1/4

In simple words

Imagine a photo slowly covered in sand until you cannot see anything. Now imagine a friend who has practised brushing sand away so often that, even from a pile of sand, they can brush out a picture. A diffusion model is that friend.

Most well-known image generators today are based on diffusion models, or close relatives of them. The idea is surprising: to learn to make images, the model learns to clean noisy ones.

Training: learn to remove noise

1
Take a real image
For example, a photo of a Rangoli pattern with its caption.
2
Add noise
Add random static. Sometimes a little, sometimes so much the image is gone.
3
Ask the model to guess
Show it the noisy image and the noise level. It predicts the noise that was added (or, equivalently, the clean image).
4
Check and correct
Compare the guess with the real answer and adjust the model a little.
5
Repeat many times
Over a huge number of images and noise levels, the model gets good at cleaning.

Generation: start from pure noise

To make a new image, start with pure random static. Ask the model: "what noise is in here?" Remove a bit of it. Now the static is slightly less random. Repeat. Over many steps, shapes, then colours, then fine details appear. Early steps decide the overall layout; later steps add texture.

Step	What the picture looks like
Start	Pure TV static
Early	Blurry blobs: a dark area at the bottom, light at the top
Middle	Rough shapes: a road, trees, a sky
Late	Details: leaves, road markings, light on a scooter

Text guidance: steering with your words

The model also receives your prompt, turned into embeddings by a text encoder (often a CLIP-style one). At each step the model predicts the noise twice: once with your prompt and once without. It then pushes further in the direction of the "with prompt" guess. This trick is called classifier-free guidance.

Higher guidance: follows the prompt more closely, but images can look harsh or less natural.
Lower guidance: more natural and varied, but may drift from what you asked.
Negative prompt (in some tools): replaces the "without prompt" guess with things to avoid, such as "blurry".

Latent diffusion: work on a small sketch, not every pixel

Cleaning every pixel of a large image many times is slow. Latent diffusion uses a separate network, an autoencoder, to squeeze the image into a much smaller grid of numbers called a latent. Diffusion happens on this small latent. At the end a decoder expands it back into a full image. Stable Diffusion is a well-known open example of this approach.

Note

The field moves fast. Some newer image models replace the older denoising network with a transformer, some use a close cousin of diffusion called flow matching, and some generate images as tokens one by one like text. The "clean noise step by step" intuition still helps you understand most of them.

What this explains

Why the same prompt gives different images: each run starts from different random noise. Tools often call this the "seed".
Why fine details used to go wrong: hands, text on signs and small repeated patterns need precise detail that is easy to get wrong. Newer models do better, but check.
Why copying can happen: if an image appeared many times in training, the model can learn to reproduce it closely. This is part of the copyright debate.

Watch out

Making realistic images of real people without consent can harm them and may be illegal where you live. Many tools block it, but not all. Your choice matters more than the tool's rules.

Try this. If you can use a free image tool, run the same prompt three times. What changes between the three? What stays the same? That is the seed and the prompt at work.

Watch and read more

Lab

A traced, step-by-step denoising of a tiny 5-pixel image, on paper and optionally in Python.

~30 min

The problem

You will act as a tiny diffusion model on a "photo" that is only 5 pixels in a row. Each pixel is a number: 0 is dark, 1 is bright.

Your model has seen only three training images: left bar = 1 1 0 0 0, right bar = 0 0 0 1 1, middle = 0 1 1 1 0.
Start from this "noise": 0.3 0.9 1.2 0.4 -0.2.
Step 1: for each training image, add up how far each pixel is from the noise (just the size of the difference). Which training image is closest?
Move each noisy pixel halfway toward the closest image. Write down the new 5 numbers.
Repeat step 3 and 4 two more times. Round the final numbers to 0 or 1. Which image did you "generate"?
Optional: run the starter code, which does the same with a smarter weighted guess and fresh random noise. Change the seed a few times and note what comes out.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import math
import random

# Three tiny "training images": 5 pixels each, 0 = dark, 1 = bright
TRAIN = {
    "left bar":  [1, 1, 0, 0, 0],
    "right bar": [0, 0, 0, 1, 1],
    "middle":    [0, 1, 1, 1, 0],
}

def guess_clean(x, noise_level):
    # Best guess of the clean image for this tiny dataset:
    # a weighted average of the training images, with more weight
    # on the ones closer to x. A real model learns this skill instead.
    weights = {}
    for name, img in TRAIN.items():
        dist = sum((a - b) ** 2 for a, b in zip(x, img))
        weights[name] = math.exp(-dist / (2 * noise_level ** 2))
    total = sum(weights.values())
    return [sum(weights[n] * TRAIN[n][i] for n in TRAIN) / total
            for i in range(len(x))]

random.seed(7)  # change this number and run again
levels = [1.0, 0.7, 0.5, 0.3, 0.15, 0.05]
x = [random.gauss(0, 1) for _ in range(5)]  # start from pure noise
print("start      ", " ".join(f"{v:5.2f}" for v in x))

for i, level in enumerate(levels):
    clean = guess_clean(x, level)
    next_level = levels[i + 1] if i + 1 < len(levels) else 0.0
    # keep the clean guess, then add back a smaller amount of fresh noise
    x = [c + next_level * random.gauss(0, 1) for c in clean]
    print(f"noise {level:4.2f} ", " ".join(f"{v:5.2f}" for v in x))

final = [round(v) for v in x]
match = [name for name, img in TRAIN.items() if img == final]
print("final:", final, "->", match[0] if match else "something new")

Output

No output. Add print(...) to see values.

You are done when

The paper trace shows the distance to each training image at every step
Each step moves the pixels toward an image, and the final rounded result matches one training image
You noted how the result changed (or did not) when the seed changed
You wrote one sentence on why this tiny model can only reproduce its training images

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1The toy model almost always outputs one of its three training images exactly. What does this suggest about real models and copying?Reveal

A model that fits its training data perfectly would only reproduce what it has seen. Real models have far too many images to memorise them all, so they mostly learn general patterns and can combine them in new ways. But images repeated many times in training get more "weight", and can be reproduced closely. That is why removing duplicates from training data reduces copying, and why memorisation is part of the copyright debate.

Q2Why does the toy add back fresh noise at each step instead of jumping straight to the clean guess?Reveal

At high noise levels the clean guess is a blurry average of many possible images. Jumping straight to it would give that blur. Adding back a smaller amount of noise and guessing again lets the process commit gradually: first to a rough layout, then to details. This is also where variety comes from: different noise leads to different final images.

Please sign in to continue.

Questions people ask

Does the model store the training images and paste pieces together?

Not in the usual sense. It learns patterns that let it remove noise. But researchers have shown that, in some settings, models can reproduce training images very closely, especially ones repeated many times in the data. So "it never copies" is also not true.

Why does generating an image take many steps?

Each step removes only a little noise, because guessing a whole image from static in one go is very hard. Many methods now need fewer steps than early ones, but the step-by-step idea remains.

What is a seed?

A number that decides the starting random noise. The same model, prompt, settings and seed usually give the same image. Change the seed and you get a different image for the same prompt.

Are diffusion models only for images?

No. The same idea is used for video, audio and even in science, for example to design molecule shapes. Anything you can add noise to and learn to clean can be generated this way.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

How diffusion models make images

How diffusion models make images

Training: learn to remove noise

Take a real image

Add noise

Ask the model to guess

Check and correct

Repeat many times

Generation: start from pure noise

Text guidance: steering with your words

Latent diffusion: work on a small sketch, not every pixel

What this explains

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask