Replication and Reproducibility

Science is supposed to be self-correcting. When one scientist makes a claim, others can check it by repeating the experiment. In practice, this often doesn’t happen—and when it does, the results are troubling.

The Replication Crisis

In 2015, the Open Science Collaboration attempted to replicate 100 psychology studies published in top journals. Only 36% successfully replicated with statistically significant results. Effect sizes were typically half of the original.

Similar problems appear across disciplines:

• Medicine: Many drug trials fail to replicate
• Economics: Laboratory economics experiments often don’t replicate
• Cancer biology: Only 11% of landmark cancer studies replicated in one effort
• Social science: Large-scale replications find many failures

Why Don’t Studies Replicate?

Statistical Issues

• p-hacking: Analyzing data multiple ways until p < 0.05
• HARKing: Hypothesizing After Results are Known
• Small samples: Underpowered studies produce unreliable results
• Multiplicity: Testing many hypotheses without correction

Incentive Problems

• Publication bias: Journals prefer positive, novel results
• Career pressure: “Publish or perish” favors quantity over quality
• Replication isn’t rewarded: No glory in checking others’ work

Methodological Issues

• Hidden moderators: Effects that only appear under specific conditions
• Experimenter effects: Subtle influence of who runs the study
• Poor documentation: Can’t replicate what wasn’t recorded

What Can Be Done?

Pre-registration

Register your hypotheses and analysis plan before collecting data. This prevents p-hacking and HARKing.

Open Data and Code

Share your raw data and analysis code. Others can check your work and build on it.

Larger Samples

Statistical power matters. Underpowered studies are nearly worthless—they’re as likely to find the wrong sign as the right one.

Multi-lab Studies

Run the same experiment in multiple labs simultaneously. This reduces false positives from lab-specific quirks.

Meta-science

Study how science works. Identify which methods produce reliable knowledge and which don’t.

Reasons for Optimism

The replication crisis, though troubling, represents progress. We now know the problem exists. Many fields are implementing reforms. The next generation of scientists is trained to value replication.

Science can be self-correcting—but only if we make it so deliberately.

---

See also: Notes on Scientific Method