Questions: Field Experiments and Real-World Randomization

Question 1

A randomized job-training program assigns 500 people to training and 500 to a control group. Only 60% of the treatment group attends. A researcher compares outcomes for actual attendees vs. the control group to estimate training's effect. This approach:

• A. Is valid because randomization still holds for the subsample who chose to attend
• B. Re-introduces self-selection bias, because attendees likely differ systematically from non-attendees in motivation or ability
• C. Is conservative and will underestimate the true treatment effect for attendees
• D. Is the most informative approach because it measures the effect on people who actually received training

Answer: (B)

This is the key mistake that intention-to-treat analysis is designed to prevent. People who chose to attend training differ from those who didn't — in motivation, circumstances, distance to the training site, and other factors correlated with outcomes. Comparing attendees to controls reintroduces exactly the self-selection bias that randomization eliminated. Option C is half-right (it might underestimate the effect), but the fundamental problem is bias, not just attenuation. Option D tempts because 'effect on those who actually received it' sounds like what we want, but it cannot be recovered by simple comparison of compliers vs. controls.

Question 2

A vaccination field experiment randomizes villages to a high-coverage vaccination program or no program. After two years, disease rates fall in both treatment and control villages. The most likely explanation is:

• A. Differential attrition — sicker people dropped out of the control group, making it look healthier
• B. Spillovers — vaccination in treated villages reduced disease transmission to nearby control villages through herd immunity effects
• C. Regression to the mean — disease rates in the control group were unusually high at baseline
• D. General equilibrium effects — a government-wide vaccination campaign ran simultaneously

Answer: (B)

Spillovers occur when treatment in one unit affects neighboring untreated units — in this case, herd immunity reduces pathogen circulation across village boundaries. When this happens, the control group is no longer a clean counterfactual: controls benefit from the treatment, so the difference between treatment and control understates the true effect. The solution is to randomize at a larger unit (e.g., districts rather than villages) and to explicitly model or buffer against spillover. Option D (general equilibrium) is a different mechanism and would typically produce larger, nationwide effects.

Question 3

Intention-to-treat analysis compares participants based on whether they actually received the treatment, to get the most precise estimate of the intervention's real-world impact.

• T. True
• F. False

Answer: False

Intention-to-treat (ITT) analysis compares participants based on their *assigned* condition, not their actual receipt. ITT preserves the causal integrity of randomization by keeping the comparison groups as they were constituted by random assignment. Comparing by actual receipt re-introduces self-selection. The cost of ITT is that the estimated effect is diluted by non-compliers — it is a conservative estimate of the full treatment effect — but it is unbiased. The statement in the question describes the per-protocol analysis, which sacrifices causal validity for apparent precision.

Question 4

In a randomized field experiment, if dropout from the study is unrelated to the experimental condition, attrition does not threaten the validity of causal inference.

• T. True
• F. False

Answer: True

The danger of attrition is *differential* attrition — when dropout rates or dropout types differ between treatment and control groups, reintroducing selection bias into groups that were initially balanced by randomization. If attrition is random with respect to condition (equal rates and patterns across arms), the remaining samples are still comparable, and causal inference is preserved. Researchers test for differential attrition by checking whether dropout rates and baseline characteristics of dropouts differ across conditions.

Question 5

What is intention-to-treat analysis, and why does it preserve causal validity even when many assigned participants don't comply with their condition?

Model answer: Intention-to-treat (ITT) analysis estimates effects by comparing groups as defined by random assignment, regardless of whether participants actually received the treatment. Because randomization made the groups statistically equivalent before the study began, any outcome difference between assigned groups can be causally attributed to the assignment — even if many people didn't comply. Analyzing by actual receipt breaks this guarantee because compliers and non-compliers differ systematically, reintroducing confounding. ITT estimates are conservative (diluted by non-compliers) but unbiased.

The logic is that randomization is a valid instrument for treatment receipt: assignment is randomly determined, affects receipt, and affects outcomes only through receipt. This is the IV setup. ITT uses assignment directly; if you want the effect specifically for compliers, instrumental variables (using assignment as an instrument for receipt) recover a Local Average Treatment Effect (LATE). The choice between ITT and LATE depends on the policy question: ITT answers 'what happens when you roll out this program?' (realistic, since compliance will be imperfect); LATE answers 'what is the effect for those who actually take up the treatment when offered?'