Questions: Imbalanced Classification and Class Weighting

Question 1

A fraud detection model reports 99.8% accuracy on a dataset where 0.2% of transactions are fraudulent. What is the most important concern about this result?

• A. The accuracy is impressive and the model should be deployed immediately
• B. The model is overfit and needs regularization to improve generalization
• C. 99.8% accuracy is achievable by predicting 'not fraud' for every transaction — the model may be catching zero actual fraud cases
• D. The accuracy threshold is too low; fraud models require 99.99% accuracy

Answer: (C)

This is the central trap of imbalanced classification. A trivial classifier that always outputs the majority label achieves exactly the majority class frequency as its accuracy — here, 99.8%. Such a model is useless for fraud detection because it never identifies a fraudulent transaction. High accuracy on imbalanced data is a misleading metric precisely because the loss function during training rewards majority-class correctness. The right question to ask is: what is the model's recall (sensitivity) for the minority class?

Question 2

You have a medical diagnosis model with 50:1 class imbalance (healthy vs. diseased). Which intervention most directly addresses the training algorithm's bias toward predicting 'healthy' for every patient?

• A. Collecting more data from healthy patients to improve the majority-class decision boundary
• B. Assigning higher loss weights to misclassified diseased patients so the optimizer treats each missed diagnosis as seriously as many missed healthy predictions
• C. Increasing model complexity so the decision boundary can separate the classes
• D. Reducing the learning rate to allow the model to find a more balanced optimum

Answer: (B)

Class weighting directly modifies the loss function so that a minority-class error contributes proportionally more to the total loss. With a 50:1 imbalance, weighting minority errors 50× more heavily means missing one diseased patient costs as much as misclassifying 50 healthy ones — which is often the correct reflection of clinical costs. This is more targeted than data-level interventions (oversampling/undersampling) because it preserves the original data distribution while correcting the learning objective. Model complexity and learning rate adjustments do not address the fundamental class imbalance problem.

Question 3

A model that achieves 99% accuracy on an imbalanced dataset is likely performing well on the minority class.

• T. True
• F. False

Answer: False

High accuracy on imbalanced data is almost meaningless as a quality signal. If the minority class represents 1% of cases, a model that never predicts it achieves 99% accuracy trivially. Minority-class performance must be evaluated using metrics that directly measure it: precision (what fraction of positive predictions are correct), recall (what fraction of true positives are caught), F1-score (harmonic mean of precision and recall), or the precision-recall curve. Accuracy alone conflates majority-class performance with overall quality.

Question 4

Lowering the classification threshold (e.g., from 0.5 to 0.1) in a probabilistic classifier increases recall for the minority class at the cost of more false positives.

• T. True
• F. False

Answer: True

A classifier typically predicts the positive (minority) class when the estimated probability exceeds a threshold. Lowering this threshold means the model classifies more examples as positive — it catches more true positives (higher recall) but also flags more negatives incorrectly (more false positives, lower precision). This tradeoff is visualized in the precision-recall curve. The optimal threshold depends on the relative cost of false negatives versus false positives: in cancer screening, high recall (missing few cases) matters more than high precision.

Question 5

Why is accuracy a misleading metric for imbalanced classification, and what alternative metrics should be used?

Model answer: Accuracy rewards correct predictions proportionally across all classes. When one class dominates, a classifier can achieve high accuracy simply by always predicting the majority label, without ever identifying a minority-class example. Useful alternatives are precision (fraction of positive predictions that are true positives), recall (fraction of actual positives correctly identified), F1-score (harmonic mean of precision and recall), and the area under the precision-recall curve — all of which measure minority-class performance directly.

The problem is that accuracy treats every error equally regardless of which class is misclassified. In fraud detection or disease diagnosis, a false negative (missing a fraud or disease) is far more costly than a false positive. The choice of metric should reflect the asymmetry of error costs in the application. Precision-recall curves make this tradeoff explicit across all possible thresholds, letting practitioners choose the operating point appropriate for their use case.