How to do AI researchMiddleLesson 3258 min read

Reproducing a result

The best way to truly understand a paper is to make its result appear on your own computer. Start small, match one number, and log every difference.

✦ Ask the tutor🧪 Lab · 40 min📝 Lesson test
Lesson in motion

In 60 seconds

Reproducing a result

The best way to truly understand a paper is to make its result appear on your own computer. Start small, match one number, and log every difference.

1/6
In simple words
Your grandmother makes a dish you love. You follow her recipe at home, but it tastes different. Was it the salt, the pan, or the fire? You change one thing at a time until it tastes right. Now you really know the recipe. Reproducing a research result is the same.
To reproduce a result means to get the same finding the paper reports, by running the experiment again. It teaches you more than reading ever can: you meet every hidden detail the paper left out. It is also a real contribution. A careful report that says "this holds" or "this holds only under these conditions" helps the whole field.
Note
People use words like reproduce, replicate and repeat in slightly different ways. What matters is saying clearly what you did: same code or new code, same data or new data, same settings or changed settings.

Step 1: pick the right paper

  • Small: the main experiment trains or runs in hours on one GPU or a laptop, not weeks on many GPUs.
  • Open: the data is public, and ideally the authors shared code.
  • Clear: it reports a specific number you can aim for, such as accuracy on a named dataset.
  • Interesting to you: you will spend many hours with it.
If the full experiment is too big, choose one table row or a smaller setting the paper also reports. Reproducing one number well is better than failing at everything.

Step 2: find the code and data

Look for a code link in the paper, on the arXiv page, or on the authors' pages. Datasets and model weights are often on Hugging Face or a project page. Read the licence before you use them. If there is no code, you will write it from the paper, which is harder but teaches a lot.

Step 3: start tiny, then grow

  1. 1

    Run it at all

    Install, then run on 100 examples for one step. Fix every error before going bigger.
  2. 2

    Overfit a tiny batch

    Train on a few examples until the loss is near zero. If it can't, something is broken.
  3. 3

    Check the evaluation code

    Score the authors' released model or predictions, if they shared any, and compare with the paper.
  4. 4

    Run a small version

    A fraction of the data or steps, to check the trend looks right.
  5. 5

    Run the target setting

    Now spend the real compute, with more than one seed if you can.

Step 4: log everything

Keep a lab notebook: a plain text or markdown file with dated entries. Every time you run something, write what you ran, what you changed, and what came out. When a number doesn't match, this log is how you find out why.
A lab notebook entrytext
2026-02-14  run 07
Goal: match Table 2, row 3 (paper reports accuracy 84.1)
Code: authors' repo, commit abc1234
Changes from paper: batch size 16 (paper: 64) because of GPU memory,
                    used gradient accumulation x4 to keep the same effective batch
Seeds: 0, 1, 2
Result: 82.6, 83.3, 82.9  (mean 82.9)
Gap: about 1.2 below the paper
Next: check tokenizer version; paper footnote mentions lowercasing

When the number doesn't match

This is very common, and it is not automatically anyone's fault. Work through the usual causes one at a time.
Possible causeWhat to check
Random seed variationRun several seeds. Is the paper's number inside your spread?
Different data version or splitSame dataset version, same preprocessing, same test split?
Different evaluationSame metric code? Case, punctuation or tokenisation handled the same way?
Hidden settingsLearning rate schedule, warm-up, weight decay, early stopping. Check the appendix and the code's default config.
Library versionsA new library version can change defaults. Try the versions the authors list.
Hardware and precisionDifferent GPUs and lower precision can change results slightly.
Your own bugCompare your code with theirs line by line on one small example.
Do this
If you still cannot match after honest effort, write to the authors. Be polite and specific: which number, which settings, what you got, what you already checked. Many authors are happy to help, and sometimes they find a bug in their own paper. A clear gap report is useful even if the gap never closes.
Reproducing badly
  • Starting with the full, largest run
  • Changing five things, then rerunning
  • Keeping results only in your memory
  • Saying "the paper is wrong" after one failed run
Reproducing well
  • Starting with 100 examples and one step
  • Changing one thing at a time
  • Logging every run with its settings
  • Reporting the gap, what you tried, and what you're still unsure of
Real example
Arjun, a self-learner in Kochi, reproduced a small text-classification paper on a free notebook GPU. His first result was clearly below the paper. His log showed he had used a different version of the dataset. After switching, he landed within his seed spread of the paper's number. He wrote a short blog post about the version issue, and other learners used it.
Try this. Think of one paper you read recently. Answer three questions: Is the code public? Is the data public? Could one experiment run on a laptop or a free GPU in a day? If all three are yes, it is a good reproduction target.

Watch and read more

Hugging Face (datasets and models)Hugging Face · reading

Lab

A reproducibility checklist for one real paper, with a verdict on whether you could reproduce it and a first-step plan.

~40 min

The problem

Take the paper from the previous lab, or any small AI paper with a clear number to match.

  1. Copy the checklist below and answer every line with yes, no or unclear, plus where you found it (section, table, appendix, code file).
  2. Choose one target number: which table, which row, what value.
  3. Estimate the compute: can it run on a laptop or a free GPU? If not, what smaller setting in the paper could you try?
  4. Write your first three steps (for example "run eval on authors' released model", "overfit 10 examples").
  5. If you have time, do step 1 and write a lab notebook entry for it.
Starter codetext
REPRODUCIBILITY CHECKLIST
Paper:
Target number (table/row/value):

Code released?                        yes / no / unclear  - where:
Exact data and version named?         yes / no / unclear  - where:
Train / validation / test split given? yes / no / unclear - where:
Preprocessing described?              yes / no / unclear  - where:
All hyperparameters listed?           yes / no / unclear  - where:
How settings were chosen (val set)?   yes / no / unclear  - where:
Number of seeds and spread reported?  yes / no / unclear  - where:
Evaluation metric code available?     yes / no / unclear  - where:
Library versions listed?              yes / no / unclear  - where:
Compute used (hardware, hours)?       yes / no / unclear  - where:
Licence allows my use?                yes / no / unclear  - where:

Can I run it on my compute?
Smaller setting if not:
First three steps:
1.
2.
3.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1You get a result above the paper's number. Is that good news?Reveal

Be careful. It could be seed luck, a different data version, or a bug such as test data leaking into training. Treat an unexpectedly good result with the same care as a bad one: check the evaluation, the split and several seeds before you believe it.

Q2The paper lists no library versions and the code fails with today's versions. What do you do?Reveal

Look at the date of the code's last commit and try library versions from around that time, in a separate environment so you don't break other projects. Record every version you end up with in your log and in a requirements file. This detail is useful to share, because others will hit the same problem.

Please sign in to continue.

Questions people ask

Is reproducing a paper "real research"?

Yes. Checking whether results hold is a core part of science. It also builds exactly the skills you need for original work. Many researchers find their first original idea while reproducing someone else's result, because they notice something the paper missed.

How close is "close enough"?

There is no fixed rule. A useful test: run several seeds and see whether the paper's number falls inside the range you get. If it does, you have likely reproduced it. If it is far outside, look for a cause, and report the gap honestly.

Can I use the authors' code, or must I write my own?

Both are useful and they answer different questions. Running their code checks that the reported number can be reached. Writing your own checks that the paper describes the method fully. Say clearly which one you did.

What if the authors don't reply?

That is common, as people are busy. Write up what you found anyway, with your log, settings and code. State the gap and your best guess of the cause, marked as a guess.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress