Questions: Sensitivity Analysis: Robustness to Unmeasured Confounding

Question 1

A researcher estimates that a tutoring program raises test scores by 8 points after matching students on income, prior grades, and school quality. A reviewer asks: 'What about student motivation — more motivated students might both seek tutoring and score higher regardless?' Which response best demonstrates sensitivity analysis thinking?

• A. Motivation is unmeasured, so we cannot draw any causal conclusion from this study
• B. We estimate that an unmeasured confounder would need to roughly triple the odds of receiving tutoring while also raising baseline scores by 5 points to eliminate our result — we judge that implausibly strong
• C. Since motivation correlates with tutoring, we should re-run the study controlling for it
• D. The 8-point effect is large enough that motivation alone couldn't plausibly explain it away

Answer: (B)

Sensitivity analysis responds to the threat of unmeasured confounding not with despair or dismissal, but by quantifying the threshold: how strong would the confounder need to be to overturn the conclusion? The correct response names the specific magnitude, then argues whether confounding that large is plausible. Option A overcorrects (treating all observational evidence as worthless). Option D undercorrects (asserting robustness without quantifying it). Option C misses the point entirely — if motivation is unmeasured, you cannot 'control for it' in a future study without first measuring it.

Question 2

A study reports its Rosenbaum sensitivity analysis result as Γ = 2.5. What is the correct interpretation?

• A. The study's result could be explained by a confounder present in 2.5% of the sample
• B. An unmeasured confounder would need to increase the odds of treatment assignment by a factor of 2.5 — holding all measured covariates constant — to eliminate the statistical significance of the result
• C. The treatment effect is 2.5 times larger than any measured covariate's effect
• D. The confidence interval spans 2.5 units on either side of the point estimate

Answer: (B)

Gamma parameterizes unmeasured confounding as an odds ratio for treatment assignment. Γ = 2.5 means that to explain away the result, an unmeasured confounder would need to make matched individuals 2.5 times more likely to receive treatment — after accounting for all measured covariates. Higher Γ means greater robustness: the unmeasured confounder required to nullify your result is increasingly implausible. This framing turns the question over to domain expertise: is there a plausible variable that strongly predicts both treatment and outcome but wasn't measured?

Question 3

An unmeasured variable that strongly predicts who receives treatment but has no relationship to the outcome cannot confound the estimated treatment effect, even though it is unmeasured.

• T. True
• F. False

Answer: True

Confounding requires that a variable affect both treatment assignment AND the outcome. A variable that only predicts treatment creates imbalance between groups on that variable, but since it doesn't affect outcomes, it produces no bias in the estimated treatment effect. Similarly, a variable that affects outcomes but is balanced between treatment and control cannot confound either. This is not merely definitional — it has practical implications: sensitivity analysis focuses on unmeasured variables that could plausibly affect both, not just any unmeasured variable.

Question 4

Conducting sensitivity analysis on an observational study's results weakens the causal claim by acknowledging that confounding might exist, effectively converting a causal finding into a correlational one.

• T. True
• F. False

Answer: False

Sensitivity analysis doesn't weaken a causal claim — it makes it honest and defensible. The output remains a conditional causal claim: 'If unmeasured confounding is weaker than Γ = 2.5, the effect is causal.' This is not correlation. All observational causal inference is conditional on unverifiable assumptions; sensitivity analysis makes those conditions explicit and evaluable, which strengthens the argument rather than weakening it. Pretending the no-confounding assumption is definitely satisfied — without sensitivity analysis — is the weaker, less honest practice.

Question 5

What does sensitivity analysis actually accomplish, and what is the key conceptual shift it demands from researchers?

Model answer: Sensitivity analysis quantifies the threshold at which unmeasured confounding would overturn a conclusion — it does not eliminate or verify the no-confounding assumption. The key shift is treating causal conclusions as conditional rather than absolute: the output becomes 'IF unmeasured confounding is weaker than X, THEN the effect is real,' paired with a substantive argument about whether X is plausible. This converts a hidden, unexamined assumption into an explicit, debatable parameter.

Without sensitivity analysis, the no-unmeasured-confounding assumption lurks implicitly — researchers proceed as if it is satisfied without stating or defending it. With sensitivity analysis, the assumption becomes quantified: you can ask domain experts whether any plausible unmeasured variable could increase treatment odds by a factor of 3. This is a tractable question that marshals subject-matter knowledge and makes the causal argument transparent and contestable. The goal is not to fix the problem (nothing can) but to be honest about its magnitude and argue that it is insufficient to overturn your conclusion.