Questions: Machine Learning Applications in Social Science

Question 1

A criminal justice agency builds an ML model achieving 89% accuracy at predicting recidivism, using features including prior convictions, neighborhood, and race. A critic argues the model is problematic even given its accuracy. What is the most substantive articulation of this critique?

• A. ML models are never accurate enough for high-stakes decisions — a 99% threshold should be required
• B. The model encodes historical patterns of discriminatory policing and sentencing; deploying it for parole decisions embeds those patterns into future outcomes and perpetuates the discrimination
• C. Parole decisions should always be made by human judges regardless of algorithmic accuracy
• D. The model violates privacy by using demographic data in its predictions

Answer: (B)

The core issue is that ML models learn correlations from historical data. If race predicts recidivism in the training data, it is because of historical patterns of discriminatory policing and sentencing — not because race causes reoffending. A model trained on this data learns the historical discrimination. Deploying it for future decisions bakes those patterns into parole decisions, disproportionately penalizing groups that were already over-policed. High accuracy on historical data does not equal fairness or legitimacy for future decision-making.

Question 2

A social scientist uses a gradient-boosted tree model to predict voter turnout with 91% accuracy and announces 'we now understand why people vote.' What is wrong with this conclusion?

• A. 91% accuracy is below the threshold required for scientific conclusions about social behavior
• B. Gradient-boosted trees cannot be applied to binary outcomes like voting
• C. High predictive accuracy demonstrates that the model identifies causes, not just correlates, of voting
• D. Predictive accuracy means the model finds patterns that forecast outcomes — it says nothing about the causal mechanisms that produce those outcomes

Answer: (D)

Prediction and causal explanation are fundamentally different. A model may achieve high accuracy by exploiting proxies (neighborhood, income, age) that correlate with voting without identifying what causes any individual to vote or abstain. Correlation can arise from confounding, reverse causation, or historical patterns. Understanding 'why people vote' requires theoretical frameworks and causal inference methods (experiments, instrumental variables, regression discontinuity) — not predictive accuracy. ML tells you *where* patterns are; causal inference tells you *why*.

Question 3

A machine learning model that achieves high overall accuracy on a prediction task can be assumed to be performing fairly across most demographic subgroups.

• T. True
• F. False

Answer: False

High overall accuracy can mask systematically unequal error rates across subgroups. A recidivism prediction model might have 90% overall accuracy while having a much higher false positive rate for Black defendants than white defendants — incorrectly flagging innocent people as high-risk at disparate rates. Overall accuracy aggregates across all cases; fairness requires examining error rates by subgroup. Equal overall accuracy is compatible with deeply unequal false positive or false negative rates across demographic categories.

Question 4

In social science, ML methods are most appropriately used for pattern discovery and generating hypotheses to investigate further, rather than as the final word on causal mechanisms.

• T. True
• F. False

Answer: True

ML excels at finding non-linear patterns in large datasets — it can identify that certain configurations of variables predict an outcome in ways that no researcher thought to specify in advance. This makes it a powerful tool for hypothesis generation: 'this unexpected cluster of features predicts protest participation — why might that be?' But answering 'why' requires causal inference methods. ML's role is as a first-pass pattern detector that tells you where to direct causal investigation, not a replacement for theory and causal design.

Question 5

A researcher builds an ML model using neighborhood characteristics to predict mortgage default with 90% accuracy. Why can't this model be taken as evidence that neighborhood characteristics *cause* mortgage default?

Model answer: Because prediction and causal inference are different. The model learns correlations — it finds that certain neighborhood features co-occur with default. But correlation can arise from confounding variables (neighborhood is correlated with income, which causally affects ability to pay), historical discrimination built into the data (redlining produced the correlation), or reverse causation. The model cannot distinguish between 'living in this neighborhood causes default' and 'the factors that put people in this neighborhood also affect their ability to pay.' Establishing causation requires controlling for confounds through experimental or quasi-experimental design.

This is the central limitation of ML in social science: it is a pattern-detection engine, not a causal inference engine. A model can achieve high predictive accuracy by exploiting proxy variables that are correlated with outcomes without being on the causal path. Race, ZIP code, and neighborhood all predict many social outcomes — but they predict because of historical patterns of discrimination and structural inequality, not because they cause outcomes directly. Using predictive models for causal claims (or for policy intervention) requires confronting this distinction explicitly.