Questions: Synthetic Control Methods for Policy Evaluation

Question 1

A researcher wants to estimate the effect of a tax reform implemented in one state in 2010, using 40 other states as potential controls. Pre-treatment trends in the treated state are not well matched by any single control state. Which method is most appropriate?

• A. Difference-in-differences using the single most similar state as the control
• B. A synthetic control constructed as a weighted average of control states that together match the treated state's pre-reform path
• C. A randomized controlled trial comparing the treated state to randomly assigned controls
• D. Ordinary least squares regression with state fixed effects, ignoring the pre-trend mismatch

Answer: (B)

When no single control unit matches the treated unit's pre-treatment trajectory, synthetic control constructs a counterfactual from a weighted combination of multiple donor units. The weights are chosen to minimize pre-treatment discrepancy. This is precisely the setting synthetic control is designed for: one treated unit, many potential controls, poor single-unit match. Difference-in-differences would be inappropriate here because the parallel trends assumption fails when pre-treatment trends diverge. A randomized trial is impossible since the policy already happened.

Question 2

In a synthetic control study, the pre-treatment fit between the treated unit and its synthetic counterpart is very poor. What does this imply?

• A. The post-treatment gap is likely to overestimate the true effect, since the synthetic control overshoots
• B. The counterfactual is less credible — if the synthetic control could not track the treated unit before treatment, there is less reason to trust it afterward
• C. The method failed; the researcher should add more pre-treatment periods until the fit improves
• D. The policy had no effect, since the poor fit shows the treated and control units were fundamentally different

Answer: (B)

The credibility of synthetic control rests entirely on the quality of the pre-treatment fit. The identifying assumption is that the synthetic control would have continued on the same path as the treated unit absent treatment. If the synthetic control could not replicate the pre-treatment path — despite being specifically optimized to do so — there is no basis for trusting that it captures the counterfactual trajectory post-treatment. A poor fit does not mean there was no effect; it means the evidence for any estimated effect is weak. The researcher should report this honestly and be cautious about causal claims.

Question 3

Synthetic control inference typically uses placebo tests rather than standard t-tests because there is only one treated unit.

• T. True
• F. False

Answer: True

A t-test requires estimating the sampling distribution of the estimated effect — which requires multiple treated units to see how much the estimate would vary across samples. With a single treated unit, there is no sampling variation to estimate. Placebo tests solve this by running the same synthetic control exercise for every control unit as if it had been treated. The resulting distribution of 'fake' gaps provides a reference: if the real treated unit's post-treatment gap is much larger than the placebo gaps, it is unlikely to be noise. This is conceptually analogous to computing a p-value, but derived entirely from the data at hand.

Question 4

A tight pre-treatment fit in a synthetic control study proves that the estimated post-treatment gap reflects a real causal effect.

• T. True
• F. False

Answer: False

Pre-treatment fit is necessary for credibility but not sufficient for proof. A tight pre-treatment match increases confidence that the synthetic control is a valid counterfactual, but the causal inference still rests on the untestable assumption that the match would have continued absent treatment. Confounding events that affected the treated unit but not the donor pool, extrapolation risk, and violations of the 'no interference' assumption can all produce misleading estimates despite perfect pre-treatment fit. Inference is strengthened by combining tight pre-treatment fit with placebo tests and qualitative reasoning about alternative explanations.

Question 5

Why do synthetic control researchers use placebo tests for inference, and how do these tests work?

Model answer: Placebo tests substitute each control unit as if it were the treated unit: a synthetic version is constructed for it using the remaining controls, and the post-'treatment' gap is measured. This generates a distribution of gaps under the null hypothesis of no effect. If the actual treated unit's post-treatment gap is much larger than almost all of the placebo gaps, this is evidence that the effect is real rather than noise. Placebo tests work because the distribution of gaps for units that were never treated captures how large gaps can be due to chance alone.

The key insight is that placebo tests use the structure of the data itself to simulate a null distribution — which is exactly what a p-value does in frequentist testing, but derived from the donor pool rather than from repeated sampling. This is necessary because with one treated unit, there is nothing to compute a standard error from. Researchers typically display all placebo and real gaps in a single time-series plot, making the inference transparent and visual rather than reduced to a single p-value.