Questions: Natural Experiments and Quasi-Experimental Design

Question 1

A researcher studies the effect of air pollution on childhood asthma by comparing children living near a highway to those living far away. A colleague argues this is confounded. Which of the following would best approximate a natural experiment?

• A. Randomly assign children to neighborhoods near or far from a highway
• B. Compare children near highways built after a zoning change made for traffic reasons unrelated to neighborhood health
• C. Match children on socioeconomic status before comparing pollution exposure
• D. Use a very large sample to reduce confounding through statistical averaging

Answer: (B)

A natural experiment requires exogenous variation — assignment driven by forces unrelated to individual health characteristics. A zoning change made for traffic or political reasons (not health reasons) creates variation in highway proximity that mimics random assignment: children near the new highway were not there because of health-related factors. Option A would be a randomized trial. Matching and large samples reduce but do not eliminate confounding from unmeasured factors.

Question 2

A regression discontinuity design uses an income threshold: households just below receive a health subsidy; those just above do not. The validity of this design rests on which assumption?

• A. Households on both sides of the threshold were randomly selected from the population
• B. Households just below and just above the threshold are similar in all other health-relevant characteristics
• C. The threshold was set without knowledge of household income levels
• D. The same households appear on both sides of the threshold at different time points

Answer: (B)

A regression discontinuity exploits the assumption that households just below and just above the threshold are essentially identical in all characteristics except their eligibility status. Far from the threshold, many confounders differ between groups. Near the threshold (e.g., $30,100 vs. $29,900 income), the comparison approaches the cleanliness of random assignment. This 'local similarity' is the identifying assumption — it must be argued from context and tested with observable characteristics, not proven.

Question 3

Natural experiments can provide causal evidence comparable to randomized controlled trials when the source of exposure variation is exogenous and unrelated to individual risk factors.

• T. True
• F. False

Answer: True

This is the core logic of natural experiments. When exposure is determined by forces external to individuals — policy cutoffs, geographic accidents, weather shocks, lottery assignments — the exposed and unexposed groups are comparable in the way that random assignment creates comparability. The key is that individuals did not select into exposure based on their own characteristics, so the exposure-outcome comparison is not confounded by self-selection.

Question 4

Because natural experiments exploit exogenous variation rather than self-selection, their causal estimates automatically generalize to the full population of interest.

• T. True
• F. False

Answer: False

Natural experiments often estimate a Local Average Treatment Effect (LATE) — a causal effect specific to the subpopulation whose exposure was actually changed by the exogenous variation. A policy cutoff provides causal evidence about people near the threshold, not necessarily those far from it. External validity — whether results generalize beyond the specific setting, population, and magnitude of the intervention — is a separate question that requires its own argument.

Question 5

What does 'exogenous variation' mean in the context of natural experiments, and why is it the key requirement for supporting causal inference?

Model answer: Exogenous variation is variation in exposure driven by forces external to the individuals being studied — forces unrelated to their health status, behaviors, or risk factors. Policy decisions, geographic boundaries, weather events, and historical infrastructure choices are examples. It is the key requirement because causal inference requires comparing otherwise-identical groups that differ only in their exposure. When exposure is chosen by individuals (endogenous variation), exposed and unexposed groups systematically differ in ways that confound the relationship. Exogenous variation breaks this link: the exposure was determined by something outside the individuals, so it is not systematically related to their characteristics, approximating the exchangeability that randomization creates.

John Snow's cholera study illustrates this: which water supplier a household had was determined by historical infrastructure decisions made before cholera's transmission was understood — not by household health behavior. That historical accident created exogenous variation in water quality, making the comparison between supplier customers nearly as clean as a randomized comparison.