Questions: ROC Curves and AUC Metrics

Question 1

A fraud detection model achieves 99.9% accuracy on a dataset where only 0.1% of transactions are fraudulent. What does this tell us about the model's AUC?

• A. The model's AUC must be close to 1.0, since high accuracy implies strong discrimination
• B. We cannot determine the AUC from accuracy alone — a model that predicts 'not fraud' for every transaction achieves 99.9% accuracy but has AUC of 0.5
• C. The model's AUC is guaranteed to exceed 0.5, since it outperforms a random baseline
• D. AUC and accuracy always agree on imbalanced datasets

Answer: (B)

A model that always predicts 'not fraud' achieves 99.9% accuracy on this dataset — equal to the class imbalance ratio — while ranking every example identically and thus having AUC = 0.5 (no better than chance). AUC measures ranking quality (does the model assign higher scores to positives than negatives?), not raw correctness. High accuracy on imbalanced data tells you almost nothing about discrimination ability. This is the central reason ROC-AUC exists.

Question 2

A classifier achieves AUC = 0.85. Which interpretation is correct?

• A. The model correctly classifies 85% of examples regardless of threshold
• B. If you randomly pick one positive and one negative example, the model assigns a higher score to the positive one 85% of the time
• C. The model achieves 85% sensitivity at the threshold that maximizes accuracy
• D. The model achieves 85% precision across all thresholds

Answer: (B)

AUC has a precise probabilistic interpretation: it equals the probability that the classifier ranks a randomly chosen positive example higher than a randomly chosen negative example. This is about ranking quality, not accuracy at any specific threshold. Options A, C, and D all confuse AUC with threshold-specific metrics. The power of AUC is that it summarizes the model's discriminative ability across ALL possible thresholds, not just one.

Question 3

An ROC curve is constructed by varying the classification threshold and recording how the true positive rate and false positive rate change at each threshold.

• T. True
• F. False

Answer: True

This is the correct construction: sweep the threshold from maximum (classify nothing as positive) to minimum (classify everything as positive), computing TPR and FPR at each step. Each threshold gives one (FPR, TPR) coordinate on the curve. At maximum threshold, you are at (0, 0); at minimum threshold, at (1, 1). The shape of the curve between these points reveals how effectively the model trades off catching true positives versus generating false alarms.

Question 4

A model with AUC = 0.75 will achieve higher accuracy at nearly every possible threshold than a model with AUC = 0.65.

• T. True
• F. False

Answer: False

AUC measures ranking quality — the probability of correctly ordering a positive above a negative — not accuracy at any particular threshold. A model with higher AUC can still have lower accuracy than a lower-AUC model at specific thresholds, especially if the two models are calibrated differently or if class imbalance affects the accuracy comparison. AUC tells you which model is better at discrimination; threshold selection based on operating costs is a separate decision.

Question 5

Why is AUC more informative than accuracy when evaluating a classifier on an imbalanced dataset? What does each metric actually measure?

Model answer: Accuracy measures the proportion of correct classifications at a fixed threshold, which is dominated by the majority class when data is imbalanced. A model that always predicts the majority class achieves high accuracy despite being useless. AUC, by contrast, measures ranking quality: the probability that the model assigns a higher score to a random positive than a random negative. It evaluates the classifier's underlying discrimination ability across all possible thresholds, independently of class proportions, because TPR and FPR are each computed within their own class.

The key distinction is that accuracy conflates the model's predictions with the class distribution, while ROC-AUC eliminates the class distribution by normalizing within each class. This makes AUC the right metric when you care about the model's ability to separate classes — and then separately choose the threshold based on the actual cost of false positives vs. false negatives in the application.