Reading a paper in 30 minutes
You do not need to read every word of a paper to understand it. Read it in three passes, take notes in a fixed template, and learn to spot weak evaluation.
In 60 seconds
Reading a paper in 30 minutes
You do not need to read every word of a paper to understand it. Read it in three passes, take notes in a fixed template, and learn to spot weak evaluation.
The three passes
- 1
Pass 1 · about 5 minutes: the shape
Read the title, abstract, all figures and tables with their captions, and the conclusion. Skim the section headings. Ask: what problem, what idea, what is the main claim? - 2
Pass 2 · about 15 minutes: the method
Read the introduction and the method section. Understand the main idea well enough to explain it to a friend. Look at the experiments: what data, what baselines, what metric? - 3
Pass 3 · about 10 minutes, or much longer: the details
Check equations, training settings, the appendix and the limitations. Only do this for papers you will build on, reproduce or review.
Pass 1 in detail: what to look at
- Abstract: find the one sentence with the main claim. Underline it.
- Figures: the first figure often shows the whole idea in one picture. Result plots show what the authors are proudest of.
- Tables: find the row for the new method and the rows for the baselines. How big is the gap?
- Conclusion: it often states the claim more plainly than the abstract, and sometimes lists limitations.
Pass 2 in detail: understand the idea
A note-taking template
Title / link / date read:
Problem (1 line):
Main claim (1 line, my words):
Method (3 lines, my words):
Data and metric:
Baselines compared against:
Key result (from which table or figure):
What would convince me more:
Limitations (theirs + mine):
Questions I still have:
Could I reproduce it? (code? data? compute?):
Related papers to read next:Spotting weak evaluation
| Warning sign | Why it matters | Question to ask |
|---|---|---|
| Weak or old baselines | Beating a weak method proves little. | Is it compared with strong, recent, well-tuned methods? |
| Only one run, no error bars | Random seeds alone can change results. | How many seeds? What is the spread? |
| Tiny gains | A small gap may be noise. | Is the gain bigger than the run-to-run variation? |
| Tuned on the test set | The number is then too optimistic. | Were settings chosen on a separate validation set? |
| Only one dataset or benchmark | The method may not transfer. | Does it hold on other data? |
| Possible test leakage | The model may have seen the test data during training. | Could the benchmark be in the training data? |
| No ablation | You don't know which part helps. | What happens if each new part is removed? |
| Cherry-picked examples | A few good examples can hide many bad ones. | Are there failure cases and random samples? |
- Reading every word from start to end
- Trusting the abstract's claim as fact
- Copying sentences into notes
- Reading alone and never discussing
- Three passes, stopping when you have enough
- Checking the claim against the tables and baselines
- Writing the idea in your own words
- Explaining the paper to a friend or a reading group
Watch and read more
Lab
Filled-in notes on a real arXiv paper after a timed three-pass read, with a judgement of its evaluation.
The problem
Choose any AI paper on the arXiv that interests you. A short paper (under 12 pages) is best for a first try.
- Pass 1 (5 min, timer on): read title, abstract, figures, tables and conclusion. Write the main claim in one sentence.
- Pass 2 (15 min): read the introduction, method and experiments. Write the method in three sentences, in your own words.
- Pass 3 (10 min): pick one table and check its setup: baselines, number of runs, how settings were chosen.
- Fill in the template from the lesson (starter below).
- Go through the eight warning signs. For each, write "fine", "worry" or "can't tell", with one line of reason.
Paper title and link:
Pass 1 main claim:
Pass 2 method (3 sentences):
Data and metric:
Baselines:
Key result (table/figure no.):
Warning signs (fine / worry / can't tell + reason):
weak baselines:
single run / no error bars:
tiny gains:
tuned on test:
one dataset only:
possible test leakage:
no ablation:
cherry-picked examples:
What would convince me more:
Could I reproduce it?:You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1The paper reports results that beat all baselines, but the authors ran their own method with much more tuning than the baselines. Is the comparison fair?Reveal
Not fully. If the new method got many tuning trials and the baselines used default settings, part of the gap may come from tuning effort, not the idea. A fair comparison gives each method a similar tuning budget, or uses the baselines' best published settings. Note this as a limitation, and if you reproduce the paper, try tuning a baseline too.
Q2When is it acceptable to stop after pass 1?Reveal
Often. If the paper is not relevant to your question, or you only need to know that the idea exists, pass 1 is enough. Log it in one line so you can find it again. Reading deeply only what matters is how researchers keep up with a fast field.
Questions people ask
I don't understand the maths. Can I still read AI papers?
Yes. Passes 1 and 2 need the idea, not every equation. Many papers explain the idea in words and pictures. Over time, look up one piece of maths at a time when you truly need it, for example when you want to reproduce the paper.
How do I choose which papers to read?
Start from a question you care about. Read one well-cited survey or overview paper, then follow its references. Ask a senior student or teacher which papers are key in the area. It is better to read five papers well than fifty abstracts only.
Should I use an AI chatbot to summarise papers?
It can help for a first look, but check the summary against the paper, because chatbots can state wrong details confidently. Do pass 2 yourself. Understanding comes from your own effort to explain the idea, not from reading someone else's summary.
What if the paper is badly written?
That happens. Look for the code, a talk by the authors, or a blog post that explains it. If the key idea is still unclear after pass 2, write down the exact question. It may be a real gap in the paper, not a gap in your knowledge.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.