Questions: Policy Analysis and Health Impact Evaluation

Question 1

A city implements a sugar-sweetened beverage tax in 2018. A researcher compares average soda consumption in that city in 2019 to consumption in 2017 and finds a 15% decline, concluding the tax was effective. What is the primary threat to validity in this analysis?

• A. The researcher should have used a larger sample to achieve adequate statistical power
• B. Without a comparison group, there is no way to separate the effect of the tax from other concurrent trends — declining soda consumption nationally, health campaigns, or economic changes — that would have occurred anyway
• C. A 15% decline is too small to be practically meaningful for public health purposes
• D. The analysis should have measured sugar intake rather than soda consumption specifically

Answer: (B)

The core problem is the missing counterfactual: what would consumption have been in 2019 without the tax? A pre-post comparison within one jurisdiction cannot answer this. National trends, media campaigns, or changing demographics could all reduce consumption independently of the tax. Difference-in-differences addresses this by using a comparable untreated jurisdiction to estimate the counterfactual trajectory.

Question 2

A difference-in-differences study compares Medicaid expansion states to non-expansion states before and after the ACA. What is the central identifying assumption that must hold for the DiD estimate to be a valid causal effect?

• A. Expansion and non-expansion states must have identical pre-policy healthcare utilization rates
• B. In the absence of Medicaid expansion, the treated states would have followed the same outcome trend as the control states (parallel trends assumption)
• C. The policy must have been assigned randomly to states rather than through state legislative choices
• D. The sample must be large enough that any pre-existing differences between states become statistically negligible

Answer: (B)

DiD does not require identical pre-policy levels — states can differ in baseline outcomes. What it requires is that both groups would have moved in parallel over time without the intervention. If expansion states were on a steeper improving trajectory before the policy (perhaps because they had more liberal health policies generally), DiD would overestimate the Medicaid expansion effect. Researchers test this by examining pre-policy trends; significant divergence before the policy weakens the parallel trends assumption.

Question 3

Natural experiments can provide credible causal estimates of policy effects without random assignment, because variation in policy exposure driven by factors unrelated to the outcome serves as a quasi-random instrument.

• T. True
• F. False

Answer: True

When a policy is implemented based on circumstances independent of the health outcomes being studied — a legislative deadline, a close election, a geographic boundary, an eligibility cutoff — this 'as-if random' variation allows researchers to estimate causal effects. The logic is the same as in randomized trials: if assignment to the 'treatment' (policy exposure) is unrelated to background characteristics, differences in outcomes can be attributed to the policy. The credibility of the causal claim depends on the plausibility that the assignment mechanism was truly independent of confounders.

Question 4

A well-designed difference-in-differences study establishes causal policy effects without any identifying assumptions, because comparing the same population before and after a policy controls for most pre-existing differences.

• T. True
• F. False

Answer: False

DiD always rests on the parallel trends assumption: that in the absence of the policy, treated and control groups would have followed the same trajectory. This assumption is empirically testable (by examining pre-policy trends) but cannot be proven — it is a claim about the counterfactual. A pre-post comparison of the treated group alone would be even weaker; using a control group addresses time trends but only under the parallel trends assumption. No observational study design eliminates identifying assumptions entirely.

Question 5

Why do evaluators need to go beyond the average treatment effect when assessing health policy impacts? What does heterogeneity in treatment effects reveal that the average cannot?

Model answer: The average treatment effect answers 'did this policy work on average?' but conceals who benefits, who is burdened, and whether the policy is equitable. A sugar tax that reduces consumption by 8% on average may have a 15% effect in low-income households (who spend a higher income fraction on these beverages) and a 2% effect in high-income households. If those same low-income households bear most of the financial burden while receiving less health benefit, the policy is regressive. Heterogeneous effects also reveal substitution patterns — switching to untaxed alternatives — that the average masks. Equity analysis requires characterizing the distribution of effects across income, race, and geography.

The practical implication is that policy evaluation should not stop at the headline average effect. Equity and efficiency both require understanding heterogeneity: the policy may be achieving its goals for some subgroups while failing or harming others. Interaction terms, subgroup analyses, and distributional measures are methodological tools for uncovering this structure.