A pharmaceutical company presents a randomized controlled trial showing their drug reduces LDL cholesterol by 30% over 12 weeks in healthy male volunteers aged 25–40. A public health team also has a 20-year cohort study of 50,000 adults (men and women, all ages, with comorbidities) showing that a dietary pattern reduces cardiovascular mortality by 18%. For setting cardiovascular prevention policy in elderly women, which evidence is likely more informative?
AThe RCT, because randomization eliminates confounding and places it higher in the evidence hierarchy
BThe cohort study, because its population is more applicable to elderly women and its outcome (mortality) is directly clinically relevant
CThe RCT, because observational studies always have more confounding than randomized trials
DThey are equally informative, and the policy team should synthesize both with equal weight
This is the key distinction between hierarchy position and evidence quality. The RCT ranks higher in the hierarchy, but its internal validity advantage is irrelevant here because the trial population (healthy young men) and outcome (surrogate marker — LDL) don't answer the policy question. The cohort study has residual confounding, but its population matches the target (elderly women with comorbidities) and its outcome (mortality) is directly relevant. Applicability and outcome directness matter as much as internal validity.
Question 2 Multiple Choice
A systematic review pools results from 12 RCTs on a new antibiotic. All 12 trials had inadequate blinding, industry-funded outcome assessment, and high dropout rates. What is the most accurate characterization of this systematic review's evidence quality?
AHigh quality — systematic reviews always rank at the top of the evidence hierarchy regardless of included study quality
BModerate quality — the pooled sample size compensates for individual study weaknesses
CLow quality — a meta-analysis of biased RCTs produces a precise but biased pooled estimate, no better than its component studies
DIndeterminate — additional RCTs should be added until the bias risk averages out
A systematic review is only as good as its input studies. Pooling biased studies increases statistical precision (narrower confidence intervals) but does not reduce bias — it produces a precise estimate of a biased effect. This is the 'measuring a bent ruler more carefully' problem. Critical appraisal must assess each included study's risk of bias using standardized tools; the hierarchy position of the review type does not guarantee quality.
Question 3 True / False
A well-designed observational cohort study with long follow-up and directly clinically relevant outcomes can provide stronger evidence for a policy decision than a poorly-executed RCT with surrogate endpoints and a highly selected population.
TTrue
FFalse
Answer: True
This is the central insight: evidence hierarchy position and evidence quality are not the same thing. Hierarchy position describes ideal design advantages; actual quality depends on execution and applicability. A cohort study with large sample, long follow-up, appropriate population, and directly relevant outcomes may be far more informative than an RCT that was well-randomized but tested in the wrong population, used surrogate markers, or had high attrition.
Question 4 True / False
The evidence hierarchy ranks randomized controlled trials above observational studies because RCTs usually produce more externally valid results.
TTrue
FFalse
Answer: False
RCTs are ranked higher for internal validity — specifically, because randomization controls for both measured and unmeasured confounding. But RCTs often have poor external validity: highly selective eligibility criteria, artificial settings, and short follow-up can all limit applicability. Observational studies, by contrast, often capture real-world populations and long-term outcomes. The hierarchy is about causal inference, not external validity or applicability.
Question 5 Short Answer
A public health professional receives an RCT showing a drug is effective. What three dimensions of evidence appraisal should they assess before using these results to inform policy?
Think about your answer, then reveal below.
Model answer: Internal validity (was the trial conducted without bias — appropriate randomization, blinding, complete follow-up, unbiased outcome assessment?); precision (were confidence intervals narrow enough to distinguish meaningful from trivial effects?); and applicability (does the trial population, setting, dose, and outcome reflect the population and decision context for which policy is being set?).
Each dimension can independently undermine otherwise impressive results. A perfectly randomized trial of a drug in young men without comorbidities provides an unbiased estimate — but an unbiased estimate of the effect in that specific population, not in elderly women with multiple medications. Evidence appraisal is ultimately asking: 'Unbiased estimate of what, in whom, and does that answer my question?'