Questions: Classification Metrics and Evaluation

Question 1

A fraud detection model evaluated on a dataset where 0.1% of transactions are fraudulent achieves 99.9% accuracy by predicting 'not fraud' for every transaction. What is its recall for the fraud class?

• A. 99.9% — the model is correct nearly all the time
• B. 0% — it never predicts fraud, so it catches none of the actual fraud cases
• C. 0.1% — it correctly identifies the rare fraud cases
• D. Undefined — recall cannot be computed on imbalanced datasets

Answer: (B)

Recall = TP / (TP + FN). This model predicts 'not fraud' for everything, so TP = 0 — it never correctly identifies a fraud case. All actual frauds become false negatives. Recall is therefore 0/0+all_fraud = 0%. This is the classic example motivating metrics beyond accuracy: 99.9% accuracy on a 0.1%-positive dataset is achieved trivially by predicting the majority class always, while being completely useless for the actual task.

Question 2

A cancer screening model correctly identifies 95% of actual cancer cases but also flags 40% of healthy patients as potentially cancerous. How should this tradeoff be characterized?

• A. High precision, low recall — the model is conservative and misses few real cases
• B. Low precision, high recall — the model catches most real cases but generates many false alarms
• C. High precision, high recall — catching 95% of cancers while flagging 40% of healthy patients is acceptable for screening
• D. Low precision, low recall — a 40% false positive rate means the model is unreliable

Answer: (B)

Recall = TP/(TP+FN) = 95% (catches 95% of real cases — high). Precision = TP/(TP+FP) — with 40% of healthy patients flagged, there are many false positives relative to true positives, so precision is low. This tradeoff is often acceptable in initial cancer screening, where the cost of missing a true case (a missed diagnosis) is far higher than the cost of a false alarm (a follow-up test). The low precision means many patients get unnecessary follow-ups, but high recall ensures few cancers are missed.

Question 3

F1 score is the arithmetic mean of precision and recall, so it equals (precision + recall) / 2.

• T. True
• F. False

Answer: False

F1 uses the harmonic mean: F1 = 2 × (precision × recall) / (precision + recall). The harmonic mean is always ≤ the arithmetic mean and crucially penalizes extreme imbalances. If a model has precision = 1.0 and recall = 0.01, the arithmetic mean would be 0.505 — suggesting decent performance — but F1 = 0.0198, correctly signaling that the model is nearly useless. Using the harmonic mean ensures that a very low score on either metric drags the combined score down sharply.

Question 4

Using macro-averaging to evaluate a multiclass classifier on an imbalanced dataset can make the classifier appear to perform worse than weighted-averaging, even if performance on the majority class is excellent.

• T. True
• F. False

Answer: True

Macro-averaging computes the metric separately for each class and takes the unweighted mean, treating all classes as equally important. If the model performs poorly on rare classes, macro-averaging penalizes this equally alongside common-class performance. Weighted-averaging weights each class's contribution by its frequency — so the majority class dominates the score. A model that excels on a 98%-majority class but fails on 2%-minority classes will look much better under weighted averaging than macro averaging.

Question 5

A hospital is deploying a classifier to screen patients for a rare but treatable disease. Should the classifier prioritize precision or recall, and why?

Model answer: Recall (sensitivity), because the cost of a false negative — missing a patient who actually has the disease and sending them home untreated — is far higher than the cost of a false positive. Patients flagged by the screen will receive follow-up diagnostic testing, so a false positive results in an unnecessary test, not a harmful outcome. Optimizing for precision would reduce false alarms but risk missing real cases. In medical screening, the principle is: cast a wide net first, then refine with confirmatory testing.

This illustrates the core principle: metric choice depends on the asymmetric costs of different errors. When false negatives are dangerous (missed cancers, undetected fraud, failed structural safety inspections), maximize recall. When false positives are costly (wrongful arrests, unnecessary surgery, expensive follow-up procedures with their own risks), precision matters more. The F1 score is appropriate when both errors have roughly equal cost; in practice, most real-world classification problems have asymmetric costs.