In a job training RCT, 30% of participants assigned to the treatment group never attend the program. The researcher wants an unbiased estimate of the program's causal effect. Which analysis is correct?
AAnalyze only those who actually attended — this estimates the true program effect on participants
BExclude non-compliers from both groups to balance the comparison
CAnalyze all participants based on their assigned treatment group, regardless of whether they attended
DReweight participants by their probability of compliance using propensity scores
Intention-to-treat (ITT) analysis compares outcomes based on assigned group, not actual treatment received. This is the correct approach because compliance is itself a post-randomization event — people who choose not to attend differ systematically from those who do, so analyzing by receipt reintroduces selection bias, destroying the guarantee that randomization provided. ITT gives an unbiased estimate of the effect of *offering* the program (which is often the relevant policy question). Option A — the intuitive choice — produces a biased 'complier effect' that overstates the effect for the full population.
Question 2 Multiple Choice
A randomized experiment on a new teaching method doesn't measure students' prior academic achievement, family income, or motivation. Why can its conclusion that the method improved test scores still be considered causally valid?
ABecause test score improvements are always caused by teaching methods, not confounders
BBecause regression adjustment can recover causal estimates even without randomization
CBecause random assignment makes treatment and control groups statistically equivalent on all variables — observed and unobserved — in expectation, so confounders cannot explain the difference
DBecause the large sample size eliminates confounding automatically
This is the core insight of randomization. Confounding arises when groups differ systematically on variables that also affect the outcome. Randomization doesn't eliminate confounders — it makes them irrelevant by distributing them equally across groups in expectation. Even unmeasured confounders (prior achievement, motivation, family SES) are balanced through random assignment. You don't need to measure or control for them because they can't systematically favor either group. Option D confuses sampling precision with confounding elimination — a large observational study can still be confounded.
Question 3 True / False
A large observational study with a sample of 50,000 participants produces more reliable causal estimates than a well-conducted randomized experiment with 500 participants on the same research question.
TTrue
FFalse
Answer: False
Sample size addresses statistical precision (reducing sampling variance), not confounding. A large observational study can still have severe confounding bias — the estimated effect may be precisely wrong. Random assignment, even in a small experiment, eliminates confounding in expectation, making the causal estimate unbiased even if less precise. The trade-off is that small experiments have higher variance (wider confidence intervals), but this is a quantifiable, honest uncertainty. Observational studies have bias that can masquerade as precision, which is arguably worse.
Question 4 True / False
If participants drop out of an experiment at rates that differ between treatment and control groups, this differential attrition can reintroduce selection bias even when the original randomization was conducted properly.
TTrue
FFalse
Answer: True
Attrition is a post-randomization event, and its causes are often correlated with both treatment assignment and the outcome. For example, if participants who experience the treatment's side effects are more likely to drop out, the remaining treated sample systematically differs from the control sample — even though both groups were equivalent at randomization. The original unbiasedness guarantee applies to the assigned groups at baseline; selective attrition erodes it. Researchers address this by checking for differential attrition rates, analyzing data available scenarios, and reporting bounds on estimates when attrition is non-random.
Question 5 Short Answer
What is the fundamental problem of causal inference, and how does random assignment address it — without actually observing individual counterfactuals?
Think about your answer, then reveal below.
Model answer: The fundamental problem is that you cannot observe the same unit under both treatment and control simultaneously — the counterfactual (what would have happened to this person without treatment) is forever unobserved. Random assignment solves this at the group level rather than the individual level: by assigning treatment randomly, the control group becomes a valid stand-in for the treated group's counterfactual outcome. Because every observable and unobservable characteristic is distributed equally between groups in expectation, the difference in average outcomes between groups is attributable to the treatment, not to any pre-existing difference. No individual counterfactual is recovered — instead, the average treatment effect is identified by group comparison.
The philosophical payoff of this answer is that causation is fundamentally a claim about counterfactual contrast, and experiments operationalize this contrast at the group level through randomization. The key word 'in expectation' is important — with any finite sample there is residual imbalance due to chance, which is why we use statistical tests. With very small samples, this chance imbalance can be substantial; with larger samples, it becomes negligible. Power analysis determines the sample size at which chance imbalance is small enough that we can detect effects of the size that matter.