Questions: Quasi-Experimental Designs with Nonequivalent Groups

Question 1

A researcher compares job outcomes for people who voluntarily enrolled in a job training program versus people who did not enroll. Program participants show better employment outcomes. What is the strongest threat to concluding the program caused this improvement?

• A. Demand characteristics — participants knew they were being evaluated and worked harder as a result
• B. Selection bias — people who chose to enroll may have had higher motivation, better social support, or stronger baseline skills before the program began
• C. History — a regional economic boom may have happened to coincide with the program period
• D. Attrition — some participants may have dropped out of the program before completion

Answer: (B)

Selection bias is the central threat in any nonequivalent control group design. Because enrollment was voluntary, the two groups were not equivalent from the start — people who enroll in job training programs likely differ systematically from those who don't in ways that predict employment success (motivation, conscientiousness, social resources). Any post-program difference might simply reflect those pre-existing differences rather than the program's causal effect. History and attrition are real threats, but they are secondary to this fundamental nonequivalence. The only way to rule out selection bias is random assignment — which this design lacks.

Question 2

A school assigns students scoring below 70 on a placement test to a tutoring program; those scoring 71+ are not assigned. A researcher compares outcomes for students scoring 68–69 versus students scoring 71–72. Why does this comparison provide relatively strong causal evidence compared to comparing all participants to all non-participants?

• A. Because the cutoff score is arbitrary, it effectively randomizes assignment for all students in the study
• B. Because students near the cutoff are likely very similar to each other on all relevant characteristics — they nearly received the same score — making near-threshold assignment quasi-random
• C. Because the standardized test score fully controls for all pre-existing differences between students
• D. Because regression discontinuity designs eliminate all threats to internal validity just as randomized experiments do

Answer: (B)

The power of regression discontinuity (RD) designs rests on the local quasi-randomness near the threshold. A student scoring 69 and a student scoring 71 were very similar — one or two items on a test — yet ended up in different conditions. Their pre-existing characteristics (ability, motivation, background) are likely nearly identical near this boundary, unlike the full treatment and control groups which may differ broadly. This makes the comparison near the cutoff credible. However, RD estimates are local: they tell you the effect for students near the threshold, not for all students. And RD does not eliminate all threats — it only improves causal inference for the near-cutoff comparison.

Question 3

Quasi-experimental designs cannot provide credible causal evidence because they lack random assignment.

• T. True
• F. False

Answer: False

This is the key misconception the topic addresses. While quasi-experiments do have weaker internal validity than randomized experiments, they can still provide credible causal evidence when designed and analyzed carefully. Regression discontinuity designs exploit near-random assignment at sharp cutoffs. Interrupted time-series designs use stable pre-intervention trends as counterfactuals. Difference-in-differences approaches control for baseline group differences. Quasi-experiments have produced influential and credible findings in economics, public health, and psychology precisely because researchers can analyze and address known confounds, even without randomization.

Question 4

In an interrupted time-series design, the history threat refers to the possibility that some other event occurred at the same time as the intervention and was the actual cause of any observed outcome change.

• T. True
• F. False

Answer: True

History is the primary internal validity threat in ITS designs. The design uses the pre-intervention trend as the counterfactual: if outcomes were stable before and then changed sharply at the intervention point, the intervention seems causal. But if something else also changed at that moment — a concurrent policy, an economic event, a media campaign — the design cannot distinguish between them. This threat is why adding a comparison time series (a similar unit that did NOT receive the intervention) greatly strengthens ITS evidence: if the comparison series shows no change while the treatment series does, concurrent historical events become less plausible as alternative explanations.

Question 5

What is selection bias in a nonequivalent control group design, and why does it make causal inference difficult?

Model answer: Selection bias occurs when the treatment and comparison groups differ systematically on characteristics related to the outcome before the intervention begins. Because there was no random assignment, pre-existing differences between groups can produce outcome differences that look like treatment effects but actually reflect who was selected into each group.

The core logic of causal inference is: if two groups are identical before treatment and then differ afterward, the treatment likely caused the difference. Random assignment creates that pre-treatment equivalence in expectation. Without it, the groups may differ at baseline on motivation, resources, health, ability, or countless other variables. Any post-treatment difference is then ambiguous: was it the treatment, or those pre-existing differences? Quasi-experimental methods like matching, covariate adjustment, and regression discontinuity all attempt to approximate pre-treatment equivalence, but none fully replicate the guarantee that randomization provides.