Hype, reproducibility and scientific integrity
AI makes it easy to get results that look great and are not real. Learn about leakage, overfitting and fake references, and how honest disclosure and review keep science trustworthy.
In 60 seconds
Hype, reproducibility and scientific integrity
AI makes it easy to get results that look great and are not real. Learn about leakage, overfitting and fake references, and how honest disclosure and review keep science…
Trap 1: data leakage
- The same patient, sample or molecule appears in both the training and test sets.
- Test data was used to choose settings or features, and then reported as if it were fresh.
- A feature gives away the answer, such as which hospital a scan came from, when one hospital had sicker patients.
- Data is split randomly when it should be split by time, so the model trains on the future to "predict" the past.
Trap 2: overfitting
import random
random.seed(1)
# Made-up data: labels are pure coin flips, so no real pattern exists.
data = [(i, random.choice([0, 1])) for i in range(200)]
memory = dict(data) # "training" = remember every example
def predict(x):
return memory.get(x, random.choice([0, 1]))
def accuracy(test):
return sum(predict(x) == y for x, y in test) / len(test)
leaky_test = data[:50] # test items were in training
fresh_test = [(i, random.choice([0, 1])) for i in range(1000, 1050)]
print("leaky test accuracy:", accuracy(leaky_test)) # looks perfect
print("fresh test accuracy:", accuracy(fresh_test)) # about a coin flipTrap 3: results others cannot repeat
- Share code and data where you legally and ethically can.
- Record model names, versions, dates, settings and prompts.
- Report results over several runs, not only the best one.
- Compare against a simple, fair baseline.
Trap 4: fabricated references and made-up content
Honest disclosure
- "AI was used."
- Nothing said, and hope no one asks
- Hiding it because it feels like cheating
- Which tool, for which task, and how the output was checked
- Placed where the venue asks, such as methods or acknowledgements
- Taking full responsibility for the final content
Peer review and AI
Watch and read more
Lab
A clear AI-use disclosure statement for a real or practice piece of work, plus a checklist of what you verified.
The problem
Pick a piece of work where you used AI, such as a school project, lab report or assignment. If you have none, use the seed-sprouting plan from the previous lesson's lab.
- List every way you used AI: which tool, which task (for example, search, summary, code, wording, figures), and roughly how much.
- For each use, write how you checked the output.
- Write a disclosure statement of three to five sentences. Say what AI did, what you did, and that you take responsibility for the content.
- Find the AI policy of your school, college or a journal in your field, if one exists. Does your statement meet it? Change it if needed.
- Swap statements with a friend. Can they tell exactly what the AI did?
Example disclosure (edit to fit your work):
I used an AI chatbot to brainstorm possible controls for my experiment and to
suggest clearer wording for my introduction. I also asked it for references;
I checked each one, found that one did not exist, and removed it. All data
were measured by me. I wrote the analysis and conclusions myself and take
full responsibility for the content of this report.You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your friend says, "I only used AI to fix grammar, so I don't need to disclose." How would you answer?Reveal
It depends on the policy. Some institutions do not require disclosure for simple grammar or spelling tools; others ask for all use to be declared. When unsure, disclose briefly: it costs little and builds trust. Also check that the "grammar fix" did not change meaning, since AI editors can rewrite more than you expect.
Q2A published paper reports a stunning AI result but shares no code or data. How should you treat it?Reveal
With care, not dismissal. The result might be real, but no one can check it easily. Look for independent replications, a fair baseline comparison, and signs of leakage in how the data was split. Treat it as a promising claim, not a settled fact, until others can reproduce it.
Questions people ask
Is using AI to write a paper cheating?
It depends on the rules and on how you use it. Using AI to fix grammar or suggest structure is often allowed if disclosed. Letting it invent content, data or references is not acceptable. Always follow your institution's policy and disclose clearly.
How do I know if my model has leakage?
Be suspicious of results that seem too good. Check that no item, person or near-copy appears in both training and test sets. Split by time or by group when that fits the problem. Try the model on truly fresh data. Ask someone else to review your data pipeline.
Why do hosted AI models make reproducibility harder?
The provider may update or retire a model, and the same prompt can give different answers. Record the model name, version and date, save the outputs you used, and consider open-weight models when exact repeats matter.
What should I do if I find an error in my published work?
Tell your supervisor or co-authors and contact the journal. Corrections are a normal, honest part of science. Fixing a mistake openly builds trust; hiding it destroys it.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.