Questions: Model Interpretability and Explainability

Question 1

A medical AI model achieves 97% accuracy at detecting tumors from X-ray scans. Saliency maps show the model highlights a region in the corner of each image — the same region where a metal ruler used during imaging always appears. What does this scenario most directly illustrate about interpretability?

• A. Saliency maps are fundamentally unreliable and should never be used to explain medical AI decisions
• B. High accuracy guarantees that the model has learned clinically meaningful features
• C. Interpretability tools require validation against ground truth — a plausible-looking explanation can still reflect a spurious correlation rather than the model's genuine reasoning
• D. The model must be retrained on ruler-free images before any explanation can be trusted

Answer: (C)

This is the classic 'clever Hans' failure in interpretability: the model achieves high accuracy by exploiting a spurious artifact (the ruler correlates with how images were labeled) rather than tumor features. The saliency map correctly reveals what the model is doing — but 'correctly revealing a spurious strategy' is not the same as 'the explanation looks clinically valid.' Interpretability tools must be validated against known ground truth (e.g., human expert annotations of which regions are clinically relevant) to confirm they are revealing genuine reasoning. High accuracy alone is insufficient evidence.

Question 2

SHAP values are computed to explain why a specific patient's loan application was denied, citing that their debt-to-income ratio was the most influential feature for this decision. This represents which type of interpretability?

• A. Global intrinsic — it uses a transparent model structure that is interpretable by design
• B. Global post-hoc — it summarizes feature importance across the entire model
• C. Local post-hoc — it explains a single prediction using a method applied after training
• D. Local intrinsic — it uses a simple model that is inherently transparent for individual predictions

Answer: (C)

Local methods explain individual predictions ('why did *this* applicant get denied?'), while global methods explain overall model behavior ('which features matter most across all decisions?'). Post-hoc methods are applied to an already-trained model, as opposed to intrinsic methods that use a model transparent by design (like a short decision tree). SHAP applied to explain one specific prediction is local and post-hoc. Understanding this two-axis framework (local/global × intrinsic/post-hoc) helps choose the right tool for the need.

Question 3

A perfectly faithful explanation of a neural network's prediction would be at least as difficult to interpret as the model itself.

• T. True
• F. False

Answer: True

This is the core faithfulness-simplicity tradeoff. A perfectly faithful explanation would reproduce every nonlinear transformation, every weight interaction, and every layer computation — which is exactly the model. The model is already opaque, so a faithful explanation is equally opaque. Useful explanations simplify, but every simplification risks distorting what the model actually computed. This tension means 'more interpretable' and 'more faithful' are opposing directions, not the same direction.

Question 4

Interpretability methods are most valuable after a model fails in production, since there is no benefit to examining model reasoning on a well-performing system.

• T. True
• F. False

Answer: False

Interpretability is valuable before deployment as a debugging tool — it can reveal that a high-accuracy model has learned spurious correlations (like the ruler example) that will fail on slightly different data distributions. It is also valuable during development for building trust and ensuring alignment with domain knowledge. Post-failure analysis is important, but waiting until failure means the spurious strategy has already caused harm. Regulatory requirements in many jurisdictions also mandate explanations at prediction time, not just post-failure.

Question 5

Why is a perfectly faithful explanation of a neural network's prediction inherently self-defeating as a practical tool?

Model answer: A perfectly faithful explanation would have to reproduce the model's full computation — all the weights, nonlinear activations, and layer-by-layer transformations. But that is just the model itself, which is the thing we found opaque in the first place. Practical explanations must simplify the model's behavior into something a human can understand, and every simplification introduces distortion. The more interpretable the explanation, the less it can capture the model's full behavior.

This self-defeating quality is why the field focuses on 'useful approximations' rather than exact explanations. Methods like LIME (local linear approximation), SHAP (Shapley value attribution), and saliency maps all trade faithfulness for comprehensibility. The honest interpretation of any interpretability output is: 'this is how the model appears to behave in this region, under this simplifying assumption' — not 'this is exactly what the model computed.' Good interpretability practice uses multiple complementary methods and validates them against known ground truth.