Questions: Synthetic Control and Comparative Case Studies

Question 1

A researcher applies synthetic control to evaluate a California smoking prevention policy. The synthetic control closely tracks California's smoking rate for 15 years before the policy, then diverges sharply downward after implementation. The most important feature of this result for making a causal inference is:

• A. The post-intervention gap is large in absolute terms
• B. The pre-intervention fit is close, meaning the synthetic control was a valid counterfactual before the policy
• C. The donor pool includes many states with similar demographics to California
• D. The analysis was conducted using a least-squares minimization algorithm

Answer: (B)

The validity of the causal inference rests on the pre-intervention fit. If the synthetic control closely tracked the treated unit before the intervention, we have good reason to believe it represents a credible counterfactual — what would have happened absent the policy. The post-intervention divergence then estimates the treatment effect. A large gap means little without a credible pre-period match; the pre-period fit is the foundation of the method's logic.

Question 2

Why are standard frequentist hypothesis tests (p-values based on assumed sampling distributions) inappropriate for synthetic control analyses with a single treated unit?

• A. Synthetic control estimates are always biased, making hypothesis tests invalid
• B. With only one treated unit, there is no sampling distribution from which p-values can be derived in the standard sense
• C. Synthetic control requires Bayesian inference because it uses prior information
• D. Standard tests require normally distributed outcomes, which smoking rates violate

Answer: (B)

Standard frequentist inference imagines drawing many samples from a population. With a single treated unit (one state, one country), there is no such sampling distribution — we have one observation of the treatment effect, not many. This is why synthetic control uses permutation-based placebo tests instead: apply the same method to each untreated unit as if it were treated, generate a distribution of 'effects,' and compare the real treated unit's effect to this null distribution. This is honest about the small-sample nature of the analysis.

Question 3

The quality of a synthetic control analysis depends critically on how well the weighted combination of donor pool units matches the treated unit's pre-intervention trajectory.

• T. True
• F. False

Answer: True

This is the core validity requirement of synthetic control. The synthetic control is only a credible counterfactual if it reproduces the treated unit's pre-intervention behavior. Poor pre-period fit means the synthetic control is not tracking the same underlying trends, and post-intervention divergence cannot be attributed to the intervention. Researchers should report pre-period fit explicitly as a quality check.

Question 4

Synthetic control requires finding a single donor pool unit that closely resembles the treated unit across most relevant characteristics.

• T. True
• F. False

Answer: False

This is precisely what synthetic control improves upon compared to simple comparison methods. No single unit needs to resemble the treated unit — the method finds a weighted combination (the 'synthetic' control) that matches as a composite. For example, synthetic California might be 40% Texas + 35% Florida + 15% Ohio + 10% Pennsylvania. The composite does the matching work even when no individual unit is a good match alone.

Question 5

Describe the placebo test used for inference in synthetic control and explain what it establishes about the estimated treatment effect.

Model answer: A placebo test applies the synthetic control procedure to each unexposed unit in the donor pool as if it were treated — estimating a 'placebo effect' for each. These placebo effects form a reference distribution. The researcher then asks: is the actual treated unit's post-intervention gap unusually large relative to this distribution? If the real effect is in the extreme tail of the placebo distribution, that constitutes evidence against the null hypothesis of no effect. The test is honest because it uses the same small-sample data structure instead of invoking asymptotic assumptions.

With only one treated unit, there is no frequentist sampling distribution to appeal to. Permutation inference — shuffling the 'treatment label' across units — generates an empirical null distribution using the actual data structure. This is why synthetic control inference is both more credible and more limited than large-sample methods: it makes honest use of what is actually available.