Questions: Text Classification

Question 1

A fraud detection dataset contains 99.9% legitimate transactions and 0.1% fraudulent ones. A classifier that always predicts 'not fraud' achieves 99.9% accuracy. What does this reveal?

• A. The model is performing well — 99.9% accuracy is excellent for any classification task
• B. Accuracy is a misleading metric here; the model detects zero fraud while appearing to succeed
• C. The dataset must be balanced to 50/50 before any classifier can be trained
• D. This problem requires unsupervised learning because labeled fraud examples are too rare

Answer: (B)

This is the class imbalance problem. Overall accuracy is dominated by the majority class frequency, masking total failure on the class of interest. A model predicting the majority class always has recall of 0% for the minority class — the exact metric that matters for fraud detection. Precision, recall, and F1-score per class are the appropriate evaluation tools: they reveal whether the model actually learns to identify the rare class rather than simply predicting the majority. Evaluation metric selection determines whether you can tell if your model is working.

Question 2

Why does fine-tuning a pretrained language model like BERT typically require far less labeled training data than training a classifier using TF-IDF features from scratch?

• A. BERT compresses text more efficiently, so fewer examples are needed to fill its parameter space
• B. BERT's pretraining has already learned rich language representations, so fine-tuning adapts existing knowledge rather than learning from zero
• C. TF-IDF classifiers require more data because they use more model parameters
• D. BERT processes examples more data-efficiently through its attention mechanism

Answer: (B)

Pretrained language models are trained on massive corpora to develop contextual representations of words, syntax, and semantics. This general language understanding is encoded in the model's weights before any task-specific data is seen. Fine-tuning adapts this existing knowledge to a specific classification task — the model already 'knows' what 'not good' means differently from 'good.' A TF-IDF + logistic regression system starts with zero language knowledge; every linguistic pattern must be learned from task-specific labeled examples. This is why thousands of fine-tuning examples can match what tens of thousands of examples achieve with classical methods.

Question 3

Bag-of-words models discard word order entirely, yet they can still achieve reasonable performance on many text classification tasks such as spam detection and topic classification.

• T. True
• F. False

Answer: True

For many coarse classification tasks, the presence or absence of specific words carries most of the signal regardless of order. 'Mortgage,' 'refinance,' and 'urgent' appearing in an email strongly suggests spam in any arrangement. Bag-of-words representations are therefore surprisingly effective for topic classification, language identification, and spam filtering. The limitation becomes critical for tasks where meaning depends on order — sentiment analysis, negation detection, or temporal reasoning — where 'not good' and 'good' are identical to a bag-of-words model but opposite in meaning.

Question 4

Preprocessing steps like lowercasing and stop word removal usually improve text classification performance and should be applied universally.

• T. True
• F. False

Answer: False

Preprocessing decisions are task-dependent and can help or hurt performance. Removing stop words may improve efficiency for topic classification (where function words carry little signal) but destroys meaning for tasks involving negation — 'not good' becomes 'good' if 'not' is removed. Lowercasing helps generalize ('Dog' and 'dog' are the same entity) but loses information where capitalization is meaningful (proper nouns, acronyms). Modern pretrained models often benefit from minimal preprocessing because they can learn which elements matter. 'Always improve' is false; always evaluate empirically.

Question 5

Explain why overall accuracy is an insufficient evaluation metric for a text classifier trained on a severely imbalanced dataset, and what metrics should be used instead.

Model answer: When classes are imbalanced, a classifier can achieve high overall accuracy by always predicting the majority class while detecting zero examples of the minority class. Overall accuracy is dominated by majority class frequency and masks total failure on the class of interest. Per-class precision, recall, and F1-score are needed because they reveal whether the model actually learns to identify each class — especially the rare one that often carries the highest practical importance.

The stakes of missing the minority class frequently exceed the cost of majority-class errors: undetected fraud, undiagnosed disease, missed safety alerts. If recall for the minority class is 0%, the classifier is useless for its intended purpose despite high accuracy. F1-score balances precision (how often the model is right when it predicts positive) and recall (how often it catches actual positives). Reporting these per-class reveals the real picture. Additionally, metrics like the area under the precision-recall curve (PR-AUC) are often more informative than ROC-AUC for highly imbalanced problems.