Questions: Reproducibility and Replication in Epidemiology

Question 1

A researcher conducts a meta-analysis pooling 20 observational studies on whether a dietary factor causes cancer. The pooled estimate is statistically significant with very low heterogeneity. A colleague concludes this is strong evidence for a causal relationship. Which concern most directly challenges this conclusion?

• A. Low heterogeneity confirms a true underlying effect, making this a reliable causal inference
• B. Publication bias may mean the 20 published studies are a biased sample — null and negative results were never published, producing artificial consistency
• C. Observational studies are ineligible for meta-analysis; only randomized trials can be pooled
• D. Twenty studies is too few for a valid meta-analysis regardless of heterogeneity

Answer: (B)

Low heterogeneity does not rule out publication bias — it can actually result from it. If only studies finding a significant effect in the same direction were published, the meta-analysis pools a biased, homogeneous sample that overstates the true effect. A rigorous meta-analyst would check for publication bias using funnel plots or Egger's test, and would note that observational evidence — even when pooled — is not sufficient for causal inference without triangulation.

Question 2

An initial small study (n=50) reports a large protective effect of a novel supplement (OR=0.5, p=0.04). A large replication trial (n=2,000) finds a small non-significant effect (OR=0.92, p=0.3). Which explanation best captures the 'winner's curse' phenomenon?

• A. The replication trial was underpowered and missed a real protective effect
• B. The original study used a biased population that responded unusually well to the supplement
• C. Small underpowered studies can only achieve statistical significance when the effect estimate is inflated by chance; the original's large estimate was partly noise crossing the significance threshold
• D. The replication trial's null result reflects regression to a different population mean unrelated to study power

Answer: (C)

The winner's curse in science occurs because underpowered studies only detect effects when the estimated effect size is large enough to cross the significance threshold — which happens partly by chance (random noise pushing the estimate up). The original study's OR=0.5 was likely inflated. The larger trial provides a more precise estimate near the true effect size. This systematic inflation of initial findings — not bias in the replication — is what the winner's curse describes.

Question 3

Pre-registration of a study's primary outcome and analysis plan before data collection is a key open science practice that makes selective reporting detectable.

• T. True
• F. False

Answer: True

Pre-registration publicly records the hypothesis, study design, and primary outcome before data is collected. This prevents researchers from post-hoc reframing exploratory analyses as confirmatory findings, and makes selective reporting detectable: reviewers can compare the registered protocol against what was actually reported. It changes the information structure of the research process without eliminating false positives — but makes the provenance of findings auditable.

Question 4

A meta-analysis that finds consistent results across many studies with low heterogeneity is necessarily unaffected by publication bias.

• T. True
• F. False

Answer: False

Publication bias can produce artificial consistency. If only studies finding a significant effect in the same direction get published, a meta-analysis would pool a biased, homogeneous set, yielding low heterogeneity and a 'consistent' significant result that overstates the true effect. True consistency of results is a positive sign, but it must be distinguished from consistency produced by systematic suppression of null results. Funnel plot asymmetry and registration-based methods can help detect this.

Question 5

Why does the 'winner's curse' cause initial study findings to systematically overestimate effect sizes, and what study design feature makes this problem worse?

Model answer: Small, underpowered studies can only achieve statistical significance when their estimated effect size is large enough — which happens partly by chance (sampling variation pushing the estimate up). Only studies whose estimates clear the significance threshold tend to get published, so the published literature is a biased sample skewed toward inflated findings. Smaller sample sizes make the winner's curse worse: with lower power and higher variance, the bar for detection requires a larger (and more likely chance-inflated) estimate. The winner's curse therefore predicts that initial findings from small studies will be followed by smaller, more modest estimates in larger replications.

This is distinct from fraud or p-hacking — the winner's curse is a statistical inevitability when significance-based publication filters operate on underpowered studies. Meta-analyses dominated by small studies are particularly vulnerable because they amplify the bias of each constituent study rather than correcting for it.