Questions: Publication Bias and the File Drawer Problem

Question 1

One hundred independent research teams each test whether Supplement X improves memory. The supplement has absolutely no real effect. Each team uses α = 0.05. Approximately how many teams would you expect to find a statistically significant result?

• A. Zero — if the supplement truly has no effect, properly conducted studies will always show no effect
• B. About 5 teams, purely by chance due to the 5% false positive rate
• C. About 50 teams, since statistical significance is roughly a coin flip
• D. It cannot be estimated without knowing the sample sizes used

Answer: (B)

At α = 0.05, each study has a 5% probability of a false positive — finding a significant result even when the true effect is zero. With 100 independent studies, you expect approximately 100 × 0.05 = 5 false positives. If only those 5 get published and the other 95 null results go into file drawers, the literature appears to support a real effect with five independent replications. This is the mechanical origin of publication bias — not fraud, just the selective presentation of a predictable statistical artifact.

Question 2

A meta-analysis synthesizes 20 published studies on a new therapeutic intervention and estimates a medium effect size (d = 0.50). A funnel plot shows significant asymmetry — the lower-left region (small studies with small effects) is notably absent. The most likely interpretation is:

• A. The effect is definitively real; funnel asymmetry confirms that statistical power was sufficient across studies
• B. The effect size estimate is inflated because small null-result studies were not published, biasing the meta-analytic average upward
• C. The included studies used different methodologies, making meta-analysis inappropriate regardless of funnel shape
• D. The heterogeneity proves the effect varies meaningfully across subpopulations

Answer: (B)

Funnel plot asymmetry — specifically the absence of small studies with non-significant or small effects — is the classic signature of publication bias. Small studies with null results went into file drawers; the surviving studies skew positive. The meta-analytic estimate inherits this bias, inflating the apparent effect. The true effect could be much smaller or zero. Funnel plot tests (Egger's test, trim-and-fill) can partially correct for this, but they estimate what's missing rather than directly accessing unpublished data.

Question 3

A preregistered study that finds a null result is just as scientifically informative as one that finds a statistically significant effect.

• T. True
• F. False

Answer: True

Null results narrow the parameter space of plausible effects — they rule out effect sizes above a certain magnitude. They are essential for accurate estimation of true population effects and prevent false beliefs from accumulating. The publication bias against null results is a structural preference of journals and authors, not a reflection of scientific value. Preregistration makes null results visible by creating a traceable record of all studies launched, independent of outcome, allowing the scientific community to see what was actually found rather than just what was published.

Question 4

Publication bias can be fully corrected by conducting meta-analyses, since averaging across most available studies cancels out the individual-study bias.

• T. True
• F. False

Answer: False

Meta-analyses that synthesize only published studies inherit the same publication bias — they average a biased sample. If the available studies are systematically selected for significant positive outcomes, the meta-analytic effect size estimate will be inflated. Statistical corrections like trim-and-fill or PET-PEESE can partially compensate by estimating what might be missing, but they do not recover the actual unpublished data. The bias must be addressed upstream (through preregistration, registered reports, and null-result journals) rather than corrected after the fact.

Question 5

Explain how a scientific literature can systematically mislead researchers about the true magnitude of an effect even when every individual study in that literature was conducted and reported honestly.

Model answer: Selective publication creates bias at the level of the literature rather than the individual study. Each individual study may be conducted and reported with complete integrity — but if journals and authors systematically publish significant positive findings while abandoning null results, the published record is a biased sample of all research conducted. By the mathematics of false positive rates, this sample overrepresents positive outcomes even when the true effect is small or zero. Rosenthal's file drawer calculation makes this concrete: if 95 null-result studies are hidden and 5 chance-significant studies are published, the literature appears to robustly support a real effect. The distortion is systemic, not individual.

This is the conceptual heart of publication bias. Students who think 'each study was honest, so the literature is reliable' have missed the key insight: systemic selection bias can emerge from individually honest behavior. Understanding this explains why preregistration and registered reports are structural reforms, not just ethical norms.