Questions: Development Policy Evaluation and Impact Assessment

Question 1

A government cash transfer program targets households with a poverty score below 50 on a 0–100 index. A researcher wants to estimate the causal effect of the transfer on child school enrollment. Which method best exploits this design?

• A. A randomized controlled trial, randomly reassigning households above and below the threshold
• B. Difference-in-differences, comparing enrolled children before and after the program
• C. Regression discontinuity, comparing households just above and just below the score-50 threshold
• D. Instrumental variables, using household income as an instrument for treatment status

Answer: (C)

The arbitrary score threshold creates a natural experiment: households just above 50 and just below 50 are nearly identical in all observable and unobservable characteristics, but one group receives the transfer and the other does not. Regression discontinuity exploits this sharp discontinuity to estimate the causal effect at the threshold. A new RCT would require randomly re-enrolling some ineligible households (ethically problematic). DID requires a clean pre-period. IV requires a separate instrument.

Question 2

An RCT in rural Kenya finds that a microfinance program raises household income by 15%. A policymaker plans to scale it to urban Bangladesh, expecting similar effects. What is the primary methodological concern?

• A. Internal validity — the randomization may have been compromised
• B. Statistical power — the sample was too small to detect real effects
• C. External validity — effects measured in one context may not generalize to a different population and setting
• D. Attrition bias — households that left the study may have had higher incomes

Answer: (C)

External validity (generalizability) is the core concern. An RCT in rural Kenya measures the average treatment effect for that specific population, at that time, with that implementation. Urban Bangladesh may differ in financial infrastructure, social norms, credit markets, and economic shocks. What works in one context may have no effect, or even a negative effect, in another. This is not a flaw in the RCT's internal validity — within Kenya the estimate is credible — but a fundamental limit of any single-site evaluation.

Question 3

A well-designed randomized controlled trial typically provides more credible causal estimates than a quasi-experimental method such as regression discontinuity or instrumental variables.

• T. True
• F. False

Answer: False

Method quality depends on execution and assumptions, not on the method label. A poorly implemented RCT — with high attrition, contamination between treatment and control, or compliance problems — can produce less credible estimates than a well-designed regression discontinuity that cleanly exploits a sharp, arbitrary cutoff. The evaluator's job is to match the method to the question and defend the identifying assumptions. No method is automatically superior.

Question 4

The fundamental challenge in development policy evaluation is that we can never directly observe what would have happened to program participants had they not received the intervention.

• T. True
• F. False

Answer: True

This is the counterfactual problem — the core challenge of causal inference. We observe the same person either treated or untreated, never both simultaneously. All evaluation methods (RCT, RD, DID, IV) are strategies for constructing a credible counterfactual by identifying a comparison group that represents what the treated group would have looked like without treatment. The method only changes how the counterfactual is constructed; the underlying problem is always the same.

Question 5

Why is reporting that 'a program worked' insufficient for scaling decisions, and what additional information do policymakers need from an impact evaluation?

Model answer: Policymakers need to know for whom the program worked (heterogeneous treatment effects across subgroups), at what cost (cost-effectiveness), under what conditions (implementation context, institutional capacity), and with what magnitude (effect size, not just statistical significance). A program that works for urban women but not rural men, costs $500 per beneficiary, and requires a trained NGO to implement cannot simply be scaled nationally in the same form.

Average treatment effects aggregate across a heterogeneous population. A positive average could hide null or negative effects for subgroups who should be excluded. Cost-effectiveness determines whether the same resources could produce more impact elsewhere. Implementation context determines whether the program can actually be replicated — many RCTs use NGOs with exceptional management capacity that governments cannot match. All of these details are required for a policy decision about scaling, modification, or abandonment.