How to do AI researchMiddleLesson 3269 min read

Designing a clean experiment

A good experiment answers one clear question fairly. Learn hypotheses, baselines, ablations, random seeds and clean data splits, with a small Python example.

✦ Ask the tutor 🧪 Lab · 35 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Designing a clean experiment

A good experiment answers one clear question fairly. Learn hypotheses, baselines, ablations, random seeds and clean data splits, with a small Python example.

1/6

In simple words

Two friends argue about whether a new fertiliser helps. If one plants it in a sunny field and the other plants without it in a shady field, the test tells you nothing. A fair test uses the same kind of field, the same seeds and the same water. Only the fertiliser changes. A clean AI experiment is a fair test like that.

An experiment is a planned test that answers a question. In AI research the question is often "does my change make the model better, and why?". A clean experiment gives an answer you and others can trust. A messy one gives a number that looks like an answer but may be noise or a mistake.

1. Start with a hypothesis

A hypothesis is a clear, testable guess. Write it before you run anything. Also write what result would prove you wrong.

Vague

"My method is better."
"Let's see what happens with data augmentation."

Testable

"Adding back-translation data improves Hindi-to-English accuracy on our test set, compared with the same model without it."
"If the mean gain across 5 seeds is smaller than the seed spread, I will conclude it does not clearly help."

2. Choose fair baselines

A baseline is what you compare against. Use at least one simple baseline (for example, always predicting the most common class) and one strong baseline (a well-known good method). Give the baselines a fair chance: tune them with the same effort and the same budget as your own method.

3. Plan ablations

If your method has three new parts, remove each part one at a time. This is an ablation. It tells you which part actually helps. Sometimes the answer is surprising: one part does all the work and the others add nothing.

Run	Part A	Part B	Part C	What it tells you
Full method	on	on	on	The headline result
No A	off	on	on	How much A contributes
No B	on	off	on	How much B contributes
No C	on	on	off	How much C contributes
Baseline	off	off	off	Where you started

4. Keep train, validation and test separate

Training set: the model learns from this.
Validation set: you use this to choose settings, such as learning rate or when to stop.
Test set: you look at this once, at the end, to report the final number.

Danger

If you check the test set again and again while choosing settings, your test score becomes too optimistic. It no longer measures how the model does on truly new data. Also check for leakage: the same or near-duplicate examples in both train and test, or future information in the training data.

5. Run several random seeds

A random seed sets the random choices in a run: the starting weights, the data order, sometimes the data split. Change the seed and the result changes a little, even with no other change. So one run can't tell you if a small gain is real. Run several seeds, then report the mean (the average) and the standard deviation (how spread out the results are).

The example below compares a simple baseline with a small logistic regression model on made-up data. It chooses the learning rate on the validation set, touches the test set once per seed, and reports mean and standard deviation. It uses only the Python standard library.

seed_experiment.py: an experiment across seedspython

import math
import random
import statistics


def make_data(rng, n=400, noise=1.2):
    """Two overlapping clouds of points. Label 1 sits up and right."""
    rows = []
    for _ in range(n):
        y = rng.randint(0, 1)
        c = 1.0 if y == 1 else -1.0
        rows.append(((rng.gauss(c, noise), rng.gauss(0.5 * c, noise)), y))
    return rows


def split(rows, rng):
    rows = rows[:]
    rng.shuffle(rows)
    a, b = int(0.6 * len(rows)), int(0.8 * len(rows))
    return rows[:a], rows[a:b], rows[b:]   # train, validation, test


def train_logreg(train, rng, lr, epochs=20):
    w1, w2, bias = rng.gauss(0, 0.1), rng.gauss(0, 0.1), 0.0
    train = train[:]
    for _ in range(epochs):
        rng.shuffle(train)
        for (x1, x2), y in train:
            z = max(-30.0, min(30.0, w1 * x1 + w2 * x2 + bias))
            g = 1 / (1 + math.exp(-z)) - y
            w1, w2, bias = w1 - lr * g * x1, w2 - lr * g * x2, bias - lr * g
    return lambda x: 1 if w1 * x[0] + w2 * x[1] + bias > 0 else 0


def majority_baseline(train):
    guess = 1 if 2 * sum(y for _, y in train) >= len(train) else 0
    return lambda x: guess


def accuracy(model, rows):
    return sum(model(x) == y for x, y in rows) / len(rows)


def run_once(seed):
    rng = random.Random(seed)
    train, val, test = split(make_data(rng), rng)
    # Choose the learning rate on VALIDATION data only.
    best_lr = max([0.001, 0.01, 0.1],
                  key=lambda lr: accuracy(train_logreg(train, random.Random(seed), lr), val))
    model = train_logreg(train, random.Random(seed), best_lr)
    # Touch the TEST set once, at the very end.
    return accuracy(majority_baseline(train), test), accuracy(model, test), best_lr


SEEDS = [0, 1, 2, 3, 4]
base, ours = [], []
for s in SEEDS:
    b, m, lr = run_once(s)
    base.append(b)
    ours.append(m)
    print(f"seed {s}: baseline {b:.3f}  model {m:.3f}  (lr chosen on val: {lr})")

print(f"baseline: {statistics.mean(base):.3f} +/- {statistics.stdev(base):.3f}")
print(f"model:    {statistics.mean(ours):.3f} +/- {statistics.stdev(ours):.3f}")
print(f"runs per setting: {len(SEEDS)} (std is the sample standard deviation)")

Run it and look at the per-seed lines. The model's accuracy moves from seed to seed, even though nothing else changed. Even the chosen learning rate can change. That spread is exactly what one lucky run would hide.

Note

Here the seed changes both the data split and the training. In a real project you often keep the data split fixed and vary only the training seed, or vary both and say so. Either is fine, as long as you report what you did. More seeds give a better picture; with only a few, treat the standard deviation as a rough guide.

6. Write down the plan before you look at results

1
Hypothesis
One sentence, plus what would prove it wrong.
2
Setup
Data, split, metric, baselines, ablations.
3
Budget
How many seeds and settings, and how the settings are chosen (validation only).
4
Run and log
Every run in the lab notebook, including failed ones.
5
Report all of it
Mean, spread, number of seeds, and results that did not go your way.

Watch out

Running many seeds and reporting only the best one is a form of cherry-picking. So is trying many metrics and reporting only the one that looks good. Decide your metric and seed count in advance, and report every run.

Try this. Change noise=1.2 to noise=2.0 in the example and run it again. What happens to the mean and to the spread? Write one sentence about why.

Watch and read more

Lab

A small seed-variance study showing whether a change really helps, with mean, spread and per-seed differences.

~35 min

The problem

Save the starter as seed_lab.py and run it with Python 3 (standard library only). It compares two settings, A and B, of the same small model.

Run it with NUM_SEEDS = 1. Write down what you would conclude from this single run.
Change to NUM_SEEDS = 5, then 20. Write down the means, standard deviations, and the wins, losses and ties.
Now change SETTING_B to {"epochs": 30} and SETTING_A to {"epochs": 1}, and repeat with 20 seeds.
For each comparison write one honest sentence: is B clearly better, clearly worse, or unclear? Use the spread to justify it.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

# Seed-variance lab: is setting B really better than setting A?
import math
import random
import statistics

def make_data(rng, n=300, noise=1.5):
    rows = []
    for _ in range(n):
        y = rng.randint(0, 1)
        c = 1.0 if y == 1 else -1.0
        rows.append(((rng.gauss(c, noise), rng.gauss(0.5 * c, noise)), y))
    return rows

def train_and_test(seed, epochs, lr=0.05):
    rng = random.Random(seed)
    rows = make_data(rng)
    train, test = rows[:200], rows[200:]
    w1, w2, b = rng.gauss(0, 0.1), rng.gauss(0, 0.1), 0.0
    for _ in range(epochs):
        rng.shuffle(train)
        for (x1, x2), y in train:
            z = max(-30.0, min(30.0, w1 * x1 + w2 * x2 + b))
            g = 1 / (1 + math.exp(-z)) - y
            w1, w2, b = w1 - lr * g * x1, w2 - lr * g * x2, b - lr * g
    return sum((w1 * x1 + w2 * x2 + b > 0) == (y == 1) for (x1, x2), y in test) / len(test)

SETTING_A = {"epochs": 3}   # change these two settings
SETTING_B = {"epochs": 4}
NUM_SEEDS = 1               # try 1, then 5, then 20

a = [train_and_test(s, **SETTING_A) for s in range(NUM_SEEDS)]
b = [train_and_test(s, **SETTING_B) for s in range(NUM_SEEDS)]
diffs = [y - x for x, y in zip(a, b)]   # same seed, so the pair is fair

print(f"A mean {statistics.mean(a):.3f}   B mean {statistics.mean(b):.3f}")
if NUM_SEEDS > 1:
    print(f"A std  {statistics.stdev(a):.3f}   B std  {statistics.stdev(b):.3f}")
    print(f"B minus A, per seed: {[round(d, 3) for d in diffs]}")
    wins = sum(d > 0 for d in diffs)
    losses = sum(d < 0 for d in diffs)
    print(f"B better on {wins}, worse on {losses}, tied on {NUM_SEEDS - wins - losses} seeds")

Output

No output. Add print(...) to see values.

You are done when

You recorded results for 1, 5 and 20 seeds, not only the last run
Your conclusions compare the gap between A and B with the spread across seeds
You noticed that a single run can suggest a difference, or no difference, that more seeds do not support
Each conclusion is honest: "unclear" is used when the evidence is mixed

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why does the starter use the same seeds for A and B and look at per-seed differences?Reveal

With the same seed, A and B see the same data and the same starting point, so much of the random noise is shared. The per-seed difference then shows the effect of the setting more clearly than comparing two separate averages. This is called a paired comparison.

Q2With 20 seeds, B wins on 11, loses on 9. A friend says "B wins more often, so it's better". What do you say?Reveal

11 against 9 is close to what you would expect from chance if A and B were equally good. Look at the size of the differences and the spread too. The honest summary is "no clear difference" unless more seeds or a proper statistical test shows otherwise.

Please sign in to continue.

Questions people ask

How many seeds are enough?

There is no single number. More seeds give a clearer picture, but each costs compute. Many small studies use a handful. What matters most is to use more than one, report how many, and show the spread. If the gap between methods is small compared with the spread, run more seeds before claiming anything.

Isn't the standard deviation enough to prove my method is better?

It is a good first check, not a proof. If the two ranges overlap a lot, you cannot claim a clear win. For stronger claims, use the same seeds for both methods and look at the per-seed differences, or use a proper statistical test. A statistics textbook or a teacher can help you choose one.

What if I have no validation set?

Make one. Split off part of your training data before you start, and never train on it. If data is very small, cross-validation (rotating which part is held out) is a common option.

Should I report failed experiments?

In your lab notebook, always. In a paper, report the ones that matter for the reader, such as ablations that did not help or settings where your method loses. This makes your work more believable, not less.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Designing a clean experiment

Designing a clean experiment

1. Start with a hypothesis

2. Choose fair baselines

3. Plan ablations

4. Keep train, validation and test separate

5. Run several random seeds

6. Write down the plan before you look at results

Hypothesis

Setup

Budget

Run and log

Report all of it

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask