Questions: Natural Experiments and Quasi-Experimental Design

Question 1

A researcher uses cities that experienced an unexpected river flood as a natural experiment to study the effect of economic disruption on crime. What is the most critical question she must answer to establish causal identification?

• A. Whether the flood was large enough in magnitude to produce measurable economic disruption
• B. Whether she should use regression discontinuity or difference-in-differences as her estimation strategy
• C. Whether flooded and non-flooded cities were on parallel trends before the flood, and whether flooding was truly unrelated to pre-existing differences in crime risk
• D. Whether she collected a sufficiently large sample size to detect the effect

Answer: (C)

The critical issue is whether the identification assumption holds: the flooded cities must be a plausible counterfactual for the non-flooded cities — they would have followed the same trajectory absent the flood. If flooded cities differ from non-flooded ones in ways related to crime (e.g., poorer infrastructure, higher population density), the comparison is confounded. This is the parallel trends assumption for difference-in-differences. Establishing this assumption empirically and theoretically is what separates a credible natural experiment from a mere coincidence.

Question 2

A researcher observes that earnings rose more in State A than State B after State A implemented a job training program. She concludes the program caused the increase. What critical assumption is being ignored?

• A. That the sample size in State A might be too small to produce reliable estimates
• B. That both states should have received the program to enable a valid comparison
• C. That State A and State B must have been on parallel trends before the program — i.e., they would have diverged by the same amount absent the program — and this assumption must be defended, not assumed
• D. That regression discontinuity design should have been used instead of difference-in-differences

Answer: (C)

The parallel trends assumption is the identification assumption for difference-in-differences: the treated and control units (here, states) would have followed the same trajectory absent the treatment. If State A was already growing faster than State B before the program — perhaps because it was attracting industry for unrelated reasons — the comparison attributes pre-existing trends to the program. This assumption cannot be verified for the post-treatment period, but researchers typically test it by checking whether trends were parallel in the pre-treatment period.

Question 3

A natural experiment is equivalent to a randomized controlled trial because both eliminate confounding by making treatment assignment independent of individual characteristics.

• T. True
• F. False

Answer: False

This is a common and dangerous misconception. Natural experiments approximate experimental logic but are not equivalent to true randomization. Even lottery-based natural experiments require checking that the lottery was genuinely random and that no sorting occurred. RDD requires that units can't precisely control which side of the cutoff they land on. Difference-in-differences requires parallel trends. Each design requires explicit identification assumptions that must be defended empirically and theoretically — they don't inherit the guarantees of true randomization.

Question 4

Even when treatment is assigned by lottery (as in the Vietnam draft lottery), the natural experiment requires checking that the lottery was truly random and that no one could selectively avoid or obtain their assigned treatment.

• T. True
• F. False

Answer: True

True. Angrist's Vietnam draft lottery study worked because birth dates were genuinely randomly assigned draft priority AND because 'lottery number' affected earnings mainly through its effect on actual military service. Both conditions had to be checked. If wealthy individuals could reliably obtain medical deferments regardless of lottery number, the lottery number would no longer be a valid instrument — treatment receipt would be selective. Verifying randomness and checking for 'compliance' are essential steps in any lottery-based natural experiment.

Question 5

What is an 'identification assumption' in natural experiments, and why can't a researcher simply assume it is satisfied?

Model answer: An identification assumption is a claim that must hold for the natural variation to isolate a causal effect rather than reflect confounding. For difference-in-differences it is parallel trends; for RDD it is that units can't precisely manipulate which side of the cutoff they fall on; for lottery instruments it is that lottery assignment is truly random and affects outcomes only through the treatment channel. Researchers can't simply assume these hold because they are empirical claims about the world — whether flooded cities had different pre-trends, whether medical deferments were selectively obtained, whether scores were manipulated near a cutoff. Each must be argued for using pre-treatment data, robustness checks, and theoretical reasoning about the institutional context.

The whole point of a natural experiment is that the variation is 'as-if' random — but 'as-if' requires justification, not assumption. Researchers defend identification assumptions through falsifiable pre-treatment tests (parallel trends plots, balance checks), placebo tests (does the 'treatment' affect outcomes that should be unaffected?), and institutional knowledge about the mechanism that created the variation. When identification assumptions fail, the natural experiment produces biased causal estimates — potentially worse than a transparent observational comparison that acknowledges its confounders.