A meta-analysis of ten studies on a job training program finds a pooled effect size of d = 0.45 with I² = 82%. What is the most appropriate interpretation?
AThe program reliably increases earnings by 0.45 standard deviations across all contexts
BThe high I² indicates that most variation across studies reflects real contextual differences, so the single pooled estimate may be misleading — meta-regression is needed to understand why effects differ
CThe meta-analysis is invalid because the studies produced inconsistent results
DThe pooled estimate should be fully trusted because averaging across more studies always reduces error
I² measures what proportion of cross-study variation is real heterogeneity rather than sampling noise. An I² of 82% means most variation is genuine — the studies are not all measuring the same effect in the same population. Simply reporting a pooled average obscures this: the 'true' effect size likely varies by context, population, treatment intensity, or outcome measure. Meta-regression, examining whether study-level moderators predict effect size differences, is the appropriate next step. Option A overstates certainty; option C mistakes heterogeneity for invalidity; option D misapplies the precision logic.
Question 2 Multiple Choice
Why does meta-analysis weight studies by the inverse of their variance rather than giving each study equal weight?
ALarger studies are more recent and use better methods, so they deserve more weight
BInverse-variance weighting ensures that studies with smaller standard errors — which are more precise estimates — contribute more to the pooled effect size
CStudies with low variance are rare, so weighting by inverse variance increases the number of qualifying studies
DEqual weighting would overcount null results, introducing publication bias into the pooled estimate
Inverse-variance weighting is a precision-weighting scheme: studies with smaller standard errors (narrower confidence intervals, more precise estimates) are given more weight because they contain more information about the true effect. A study with n=5,000 has a much smaller standard error than one with n=50, so it should pull the pooled estimate toward its result more strongly. Option A conflates sample size with methodological quality — large studies can be methodologically weak. Option D confuses inverse-variance weighting with a bias correction.
Question 3 True / False
A meta-analysis that includes more studies is generally more reliable than one with fewer studies, because pooling more evidence brings the estimate closer to the true effect.
TTrue
FFalse
Answer: False
More studies improve precision only when they are estimating the same underlying effect under comparable conditions. When studies are highly heterogeneous (measuring different constructs, populations, or treatments), pooling more of them can distort rather than clarify the estimate. Additionally, if the additional studies are drawn from a biased literature (e.g., all published positive findings), adding them amplifies publication bias rather than correcting it. Quality and comparability of studies matter as much as quantity.
Question 4 True / False
Publication bias can cause a meta-analysis to overestimate effect sizes, because studies finding significant positive results are more likely to be published than null or negative results.
TTrue
FFalse
Answer: True
Publication bias means the published literature is a non-random sample of all conducted studies — positive results get published; null results often do not. When a meta-analysis draws only from published studies, the pooled effect size is systematically inflated. Systematic reviews combat this by searching gray literature, unpublished dissertations, and conference proceedings, and by using statistical tools (funnel plot asymmetry, Egger's test, trim-and-fill methods) to detect and correct for the underrepresentation of null results.
Question 5 Short Answer
Explain what the I² statistic measures and why a high I² value changes what you should conclude from a meta-analysis.
Think about your answer, then reveal below.
Model answer: I² measures the proportion of total variance across studies that reflects true heterogeneity — real differences in effect sizes — rather than sampling error. A low I² (e.g., 20%) suggests most cross-study variation is noise and a pooled average is a reasonable summary. A high I² (e.g., 80%) means most variation is genuine: the studies are not all measuring the same effect, so averaging them produces a number that may not accurately represent any specific context. High I² is a signal to investigate moderators via meta-regression — to understand which study-level factors (population, dosage, outcome measure) explain why effects differ.
The key shift with high I² is moving from 'what is the average effect?' to 'under what conditions is the effect large or small?' A pooled estimate with high heterogeneity is like averaging temperatures in Alaska and Florida — the number exists but doesn't describe either place well. Meta-regression replaces the average with a conditional relationship: effect as a function of moderating variables.