Questions: Meta-Analysis Methods and Heterogeneity Assessment

Question 1

A meta-analysis of 20 randomized trials on a new drug reports I² = 78% and a pooled odds ratio of 1.4 under a random-effects model. A colleague argues the result is highly reliable because it synthesizes 20 high-quality trials. What is the most important concern?

• A. 20 studies is too few to produce a valid meta-analytic estimate
• B. High I² indicates that studies are estimating genuinely different true effects; the pooled estimate averages across this variation and may not apply to any specific context or population
• C. Random-effects models are statistically inappropriate for randomized controlled trials
• D. Odds ratios cannot be validly pooled across studies from different countries

Answer: (B)

I² = 78% means approximately 78% of the observed variation between studies reflects real differences in true effects, not just sampling error. When true effects vary this substantially across studies — perhaps due to different populations, doses, follow-up periods, or co-interventions — the pooled estimate is an average of a heterogeneous distribution. It may not represent the effect in any specific setting. The number of studies (20) is actually quite large; the problem is heterogeneity, not sample size. Understanding what drives heterogeneity is more important than the headline pooled estimate.

Question 2

A funnel plot of 15 studies shows clear asymmetry: small studies cluster only on the side showing beneficial effects, with a notable absence of small studies showing null or harmful effects. This pattern most likely indicates:

• A. The large studies used less rigorous methods and should be down-weighted
• B. Publication bias, where small studies finding null or harmful effects were less likely to be published, inflating the apparent pooled effect
• C. The random-effects model was incorrectly specified, producing asymmetric weighting
• D. Clinical heterogeneity that is unrelated to publication practices

Answer: (B)

Funnel plot asymmetry of this specific pattern — small studies only on the beneficial side, absence of small null studies — is the classic signature of publication bias. Small studies with null results are less likely to be published, so the meta-analytic sample is a biased subset of all conducted research. Large studies appear more symmetrically because they are usually published regardless of result. If the pooled effect is driven by these potentially missing small null studies, the true effect may be smaller or absent. Statistical tests like Egger's test can formalize this assessment.

Question 3

A random-effects meta-analysis produces wider confidence intervals than a fixed-effects analysis of the same studies, because random-effects models account for between-study variance (τ²) in addition to within-study sampling error.

• T. True
• F. False

Answer: True

Fixed-effects models treat the only source of uncertainty as within-study sampling error. Random-effects models add a second source of uncertainty: the variance in true effects across the distribution of study populations and settings (τ²). This additional variance appropriately widens the confidence interval, reflecting greater uncertainty about where the 'average' true effect falls. The wider interval is not a weakness — it is a more honest representation of uncertainty when heterogeneity exists.

Question 4

An I² of 0% in a meta-analysis proves that most studies are estimating the same underlying true effect, making the fixed-effects pooled estimate straightforwardly valid.

• T. True
• F. False

Answer: False

I² = 0% means the observed variation between studies is no greater than expected by chance — it does not prove the true effects are identical. The Q-test (which I² is derived from) has very low statistical power when there are few studies: with only 5 or 6 studies, the test may fail to detect substantial heterogeneity. A meta-analysis with few small studies could return I² = 0% even when true effects differ meaningfully across populations. Absence of evidence for heterogeneity is not evidence of absence.

Question 5

Explain why the choice between a fixed-effects and a random-effects model in meta-analysis is a conceptual decision about the research question, not merely a statistical choice driven by the heterogeneity test.

Model answer: Fixed-effects and random-effects models answer different questions. Fixed-effects assumes one universal true effect exists across all studies — variation is just noise — and estimates that common effect. The implicit question is: 'What is the single true effect?' Random-effects assumes there is a distribution of true effects across study contexts (different populations, doses, settings) and estimates the mean of that distribution along with its spread. The implicit question is: 'What is the average effect across this population of studies?' The choice should depend on whether a single underlying truth is scientifically plausible — not on whether the Q-test reaches significance. Even if heterogeneity is non-significant, if studies differ in important ways (population characteristics, treatment protocols), a random-effects framework better represents the scientific reality.

A common error is to use fixed-effects when I² is low and random-effects when it is high, as if the statistical test should drive the modeling choice. But the model should reflect the scientific question. In most epidemiological meta-analyses, true effect heterogeneity across populations is expected, making random-effects the default appropriate framework regardless of the test result.