Questions: Replication and the Open Science Movement

Question 1

A researcher runs 20 different analyses on their dataset, finds one with p = .03, and reports it as their primary finding without mentioning the other 19. The most accurate description of this practice is:

• A. Scientific fraud, because the researcher deliberately hid negative results
• B. P-hacking, because researcher degrees of freedom inflated the false-positive rate beyond the nominal 5%
• C. Acceptable exploratory analysis, since the finding is still statistically significant
• D. Publication bias, because the journal only accepts positive results

Answer: (B)

This is p-hacking (researcher degrees of freedom): making many analytic decisions while looking at the data systematically inflates the chance of finding p < .05 by chance. It's distinct from fraud — the researcher may not even be aware they're doing it. Publication bias refers to journal-level filtering, not researcher-level analysis choices. A finding produced this way cannot be interpreted at face value as a 5% false-positive rate.

Question 2

A researcher preregisters their study and then collects data. After looking at the results, they notice an interesting pattern not in their original plan. What does preregistration allow them to do?

• A. Nothing — preregistration legally prohibits any unplanned analyses
• B. Run the exploratory analysis but label it as exploratory, so readers can calibrate their confidence
• C. Discard the preregistered hypotheses if the exploratory finding is more interesting
• D. Publish the exploratory finding as confirmatory since it came from the same dataset

Answer: (B)

Preregistration distinguishes confirmatory from exploratory analyses — it does not eliminate exploratory work. Researchers can and should report unexpected patterns, but labeling them as exploratory signals to readers that these findings require independent replication before being treated as established. Presenting exploratory findings as confirmatory is exactly what preregistration is designed to prevent.

Question 3

A replication study fails to reproduce the original finding. This proves the original study's results were false.

• T. True
• F. False

Answer: False

Failure to replicate does not automatically mean the original was wrong. Differences in sample characteristics, cultural context, operationalizations of variables, or time period can produce genuine moderating effects — the original finding may be real but context-dependent. Replication failures are important evidence that demands explanation, but the correct response is to investigate why, not to conclude the original was fabricated or simply wrong.

Question 4

Publication bias can distort a scientific literature even when every individual researcher behaves honestly and no data manipulation occurs.

• T. True
• F. False

Answer: True

Publication bias is a systemic, structural phenomenon. When journals preferentially accept positive results and researchers preferentially submit them, the published record ends up skewed — not because anyone cheated, but because thousands of honest individual decisions collectively filter out negative and null findings. The result is a literature that overstates effect sizes and effect prevalence. This is why the Open Science movement focuses on structural reforms (registered reports, preregistration) rather than just policing individual conduct.

Question 5

Why do small, underpowered studies that achieve statistical significance tend to overestimate effect sizes — a phenomenon sometimes called the 'winner's curse'?

Model answer: In an underpowered study, only the largest-by-chance results will clear the significance threshold. If the true effect is small and the sample is small, most runs of the study will produce non-significant results. The rare runs that do achieve significance are those where sampling error happened to inflate the estimated effect. This means the subset of underpowered studies that get published (because they're significant) are selected for having inflated estimates — creating a systematic overestimation in the published literature.

This is why average effect sizes in the Reproducibility Project replications were roughly half those in the originals. The originals were often underpowered, and the published results were filtered for significance, selecting for overestimates. Larger, well-powered replications regress toward the true (smaller) effect size. The winner's curse is a direct consequence of combining publication bias with low power.