Episode 12.2: The Reproducibility Crisis: Why Researchers Needed to Change
Last updated on 2026-09-08 | Edit this page
Estimated time: 19 minutes
Overview
Questions
- What’s the real difference between reproducibility and replication?
- What was the “reproducibility crisis,” and what actually caused it?
- Why did it push so many fields toward open science?
Objectives
Learners will be able to:
- Define reproducibility and replication precisely, and explain what each one verifies.
- Describe what the reproducibility crisis was, and roughly how widespread it turned out to be.
- Identify the main contributing factors: misaligned incentives, under-resourced researchers, and questionable research practices (QRPs).
- Explain why the crisis is generally seen as a sign of science working, not failing.
Think Like a Researcher
The researcher who emailed Aisha follows up a few months later:
“We used your questionnaire, but we’re stuck reproducing your analysis. How exactly did you classify households with partially covered drainage? It wasn’t in your report.”
Aisha checks her notebook. She recorded the observation, but never wrote down how she’d handled the borderline cases: a handful of households that didn’t fit neatly into her scoring system. She made a call in the moment. It felt obvious at the time. Months later, she can’t fully reconstruct her own reasoning.
Nobody is accusing Aisha of dishonesty. The second team just needs to see exactly how she got from data to conclusion, and that trail went cold.
A Wise Scholar Once Said…
“Science is the belief in the ignorance of experts.” — Richard Feynman
Feynman’s point wasn’t that experts are useless. It’s that a claim earns trust by surviving scrutiny, not by whoever said it. That’s exactly what reproducibility checks for.
Reproduction vs. Replication
These two get used interchangeably in casual conversation, and in a lot of older news coverage, but they check different things.
Reproduction takes the original researcher’s data and code, and re-runs the same analysis. Same inputs, same steps: do you land on the same numbers? Reproduction verifies the analysis.
Replication is a new, independent study of the same question: new data, ideally the same or a similar method. Do you land on a similar answer? Replication tests whether the finding generalises beyond the original sample.
If a bookshelf comes with a full set of assembly instructions (every screw, every panel, every step), anyone following them should end up with the same bookshelf. That’s reproduction: same materials, same steps, same result. Replication is more like a different carpenter building the same design from scratch, in a different workshop, and checking whether it still holds together.
Both matter. A finding that can’t even be reproduced from its own data is on very shaky ground. One that reproduces but hasn’t been replicated independently is a step further along, but still unconfirmed outside its original sample.
Two more terms worth knowing: Credibility - How much trust a finding deserves, based on how rigorously it was produced and checked. Robustness - Getting a similar answer even when small, reasonable choices in the analysis are changed.
What Was the “Reproducibility Crisis”?
Starting around 2015, a wave of reporting (in outlets like the Washington Post and The Atlantic) highlighted a problem researchers had started documenting seriously: a surprising share of published findings, especially in psychology, weren’t holding up when other teams tried to reproduce or replicate them. Two large-scale efforts, one in psychology and one in cancer biology research, each tried to replicate a batch of high-profile published studies. In both, a large proportion of the original findings failed to replicate, and even the ones that did tended to show a much smaller effect the second time around.
A widely cited 2016 survey went further, finding that a majority of researchers across fields had failed to reproduce another scientist’s results at least once, and more than half had failed to reproduce their own. A failed reproduction doesn’t automatically mean a study was wrong. But at that scale, it’s a strong signal that something about how research was being done and documented needed a second look.
What Actually Caused It?
It’s tempting to assume fraud. In reality, deliberate misconduct explains only a small share of it. Most of the problem traces back to three ordinary, often well-intentioned pressures:
Misaligned incentives: Careers are often built on the number of publications, not their long-term reliability. Novel, surprising results get published and cited far more readily than careful replications or null results, which quietly pushes researchers toward whichever analysis produces an exciting result, rather than whichever one is accurate.
Under-resourcing: Researchers are often expected to publish more, manage larger projects, and share data properly, without necessarily being given the time, training, or staff to do all of that well. Good documentation takes time that grant timelines don’t always allow for.
Questionable Research Practices (QRPs): These sit in a grey area short of outright fraud, often unintentional, but still capable of distorting results.
p-hacking (also called data dredging): running many different analyses until one produces a “significant” result, then reporting only that one. HARKing (Hypothesizing After the Results are Known): noticing a pattern in the data first, then writing it up as though it had been predicted all along.
Selective reporting: publishing only the outcomes that worked out, and quietly leaving out the ones that didn’t.
Silent post-hoc data collection: gathering extra data after an initial analysis fell short, in hopes of pushing it over the line, without disclosing that this happened.
None of these require intent to deceive. Most researchers are taught not to do them, and end up doing some version of them anyway, especially under pressure. But across an entire field, practices like these add up, and produce exactly the pattern the 2015 headlines were describing.
How Open Science Responds
Every factor above shares a common fix: make the process visible enough that it can be checked.
| Problem | Open Science Response |
|---|---|
| Methods aren’t fully described | Share a detailed protocol and materials |
| Analysis can’t be verified | Share the data and analysis code |
| Hypotheses shift quietly after seeing results | Preregister the hypothesis and analysis plan |
| Findings can’t be checked at all | Make outputs available, not just conclusions |
None of this guarantees a study is correct; researchers will always make mistakes. What it changes is how quickly those mistakes get noticed and fixed, instead of quietly shaping the next ten studies that build on them.
What Would Have Helped Aisha
A short note in her protocol, “households with partially covered drainage were scored as moderate risk,” written the day she made that call, not reconstructed from memory months later. That’s the whole fix: not more work, just documentation captured while the reasoning is still fresh.
Reflection
Think about a decision you made partway through a recent project: a judgment call that felt obvious in the moment. Did you write down why you made it? If someone asked you to justify it six months from now, could you?
Wrap-Up: A Sign Science Is Working
“Crisis” sounds like failure. It isn’t, quite. A field that discovers its own findings don’t hold up as often as assumed, and responds by changing how it documents and shares work, is doing exactly what science is supposed to do: testing its own claims and correcting course. The reproducibility crisis didn’t break trust in research. It’s a large part of why open science practices exist at all, which is where the rest of this module picks up.
Test Your Knowledge!
Challenge 1:
A team re-runs another lab’s published analysis using the exact same dataset, and gets the exact same numbers. What have they demonstrated?
A. Replication
B. Reproduction
C. A questionable research practice
D. Nothing meaningful
B. Reproduction: same data, same steps, same result. Replication would mean collecting new data on the same question.
Challenge 2:
Which of these is a Questionable Research Practice?
- A. Preregistering a hypothesis before collecting data
- B. Reporting all outcomes measured, even the ones that didn’t work out
- C. Running several analyses and reporting only the one that came out significant
- D. Sharing analysis code alongside a paper
C. This is p-hacking: running multiple analyses until one “works,” then reporting only that one.
Challenge 3:
True or False: Most reproducibility problems in the 2015 wave of research were caused by deliberate fraud.
False. Outright fraud accounts for a small share of it. Most of the problem traces back to misaligned incentives, under-resourcing, and largely unintentional questionable research practices.
Challenge 4:
A researcher notices an unexpected pattern in their data, then writes their paper as though they’d predicted it from the start. What is this practice called?
HARKing: Hypothesizing After the Results are Known.
- Reproduction re-runs the original data and analysis to verify the result; replication collects new data to test whether the finding holds up more generally.
- The reproducibility crisis refers to a well-documented pattern, especially visible from 2015 onward, of published findings failing to hold up when other researchers tried to reproduce or replicate them.
- Most of the problem traces to misaligned incentives, under-resourcing, and questionable research practices like p-hacking, HARKing, and selective reporting, not outright fraud.
- Open science responds by making the research process visible enough to check at every stage, which is why the crisis is widely seen as evidence that science self-corrects, not that it’s broken.
Inline instructor notes…
To-do: Add infographic.
💡 A study that fails to replicate isn’t automatically wrong, and a researcher whose earlier work doesn’t hold up isn’t automatically careless. The point of reproducibility isn’t to catch people out; it’s to make sure confidence in a finding is actually earned.