Questions: Meta-Analysis in Biostatistics

Question 1

A meta-analysis of 15 trials testing a drug for hypertension reports I² = 85%. What does this indicate and what are the implications?

• A. 85% of studies support the drug — it is highly effective
• B. 85% of the observed variability in effect sizes is due to genuine heterogeneity between studies rather than sampling error, suggesting the true effect likely varies across study populations or conditions
• C. The meta-analysis has 85% statistical power
• D. 85% of patients in the studies experienced a benefit

Answer: (B)

I² = 85% indicates substantial between-study heterogeneity — the studies are not estimating the same quantity. This does not mean the meta-analysis is invalid, but it means the pooled estimate represents an average across genuinely different effects. The critical follow-up is to investigate why effects vary: are certain patient populations, drug doses, or outcome definitions driving the heterogeneity? A random-effects model is appropriate here, and subgroup analyses or meta-regression should explore sources of heterogeneity.

Question 2

A funnel plot for a meta-analysis shows asymmetry, with small studies predominantly reporting large positive effects. What does this suggest?

• A. Small studies are more rigorous and therefore find larger effects
• B. Publication bias — small studies with null or negative results were likely conducted but not published, leaving an asymmetric distribution of reported effects
• C. The treatment is more effective in small studies
• D. The meta-analysis included too many studies

Answer: (B)

A symmetric funnel plot is expected when published studies represent all conducted studies: small studies scatter widely around the pooled estimate, and large studies cluster tightly. Asymmetry — particularly an absence of small studies with small or negative effects — suggests publication bias: those studies were conducted but their results were not published because they were not statistically significant or were perceived as uninteresting. Statistical tests (Egger's test, Begg's test) and trim-and-fill methods can assess and partially adjust for this bias.

Question 3

A fixed-effect meta-analysis assumes all studies share a single true effect size, while a random-effects model assumes study-specific true effects drawn from a distribution. When there is substantial heterogeneity, the fixed-effect model gives too much weight to large studies.

• T. True
• F. False

Answer: True

Under the fixed-effect model, weights are inversely proportional to within-study variance only. Large studies have small variance and dominate the pooled estimate. Under the random-effects model, weights include an additional between-study variance component, which equalizes weights across studies — even large studies carry non-trivial between-study uncertainty. When heterogeneity is substantial, the random-effects model gives relatively more weight to smaller studies than the fixed-effect model, and the confidence interval is wider, properly reflecting the uncertainty about the true distribution of effects.

Question 4

Explain why a meta-analysis that pools only published studies, without a systematic search for unpublished data, may produce a biased summary effect.

Model answer: Studies with statistically significant or positive results are more likely to be published, submitted, and accepted. If a meta-analysis includes only published studies, it oversamples positive results and undersamples null or negative results. The pooled estimate is therefore biased toward larger effects than the true average across all conducted studies. This publication bias can make an ineffective treatment appear effective or make a modestly effective treatment appear highly effective.

This is why rigorous meta-analyses use systematic review methodology: searching multiple databases, registries of clinical trials (ClinicalTrials.gov), grey literature (conference abstracts, dissertations), and contacting study authors for unpublished data. Pre-registration of systematic review protocols (PROSPERO) further protects against selective reporting of meta-analytic results.