Questions: Bias-Variance Tradeoff

Question 1

Your model achieves 2% training error and 38% test error. What is the most accurate diagnosis?

• A. High bias — the model is too simple to capture the underlying pattern in either dataset
• B. High irreducible noise — the test data is fundamentally unpredictable regardless of model quality
• C. High variance — the model has overfit to the training data, learning noise that does not generalize
• D. Underfitting — the model needs more parameters to learn the patterns in the training data

Answer: (C)

When training error is very low but test error is high, the model has learned the training data extremely well — including its noise — but that learned pattern doesn't transfer to new data. This is the hallmark of high variance (overfitting). High bias would produce high training error AND high test error, because a biased model fails to learn even the training data well. Underfitting is just another term for high bias. The gap between training and test error is the diagnostic signal: large gap = variance problem, both errors high = bias problem.

Question 2

You apply L2 (ridge) regularization to an overfit neural network. What change in the bias-variance tradeoff should you expect?

• A. Both bias and variance decrease — regularization improves the model in all respects
• B. Bias slightly increases and variance significantly decreases — you are deliberately accepting more systematic error to reduce sensitivity to training noise
• C. Bias decreases and variance increases — regularization removes constraints that were limiting model flexibility
• D. Neither bias nor variance changes — regularization only affects training speed, not generalization

Answer: (B)

Regularization works by adding a penalty that discourages large weights, effectively constraining the model's flexibility. This introduces a small systematic bias — the model can no longer perfectly fit arbitrary training data — but in exchange, the model's predictions become more stable across different training samples (lower variance). This is the explicit tradeoff regularization is designed to make: accept a small increase in bias to gain a large reduction in variance, leading to better generalization overall.

Question 3

Minimizing model bias should usually be the primary goal in machine learning, since lower bias means the model makes fewer systematic errors.

• T. True
• F. False

Answer: False

This is the most important misconception the bias-variance tradeoff corrects. A model with very low bias but very high variance can generalize poorly — it memorizes training noise and fails on new data. A model with moderate bias and low variance often generalizes much better. Total test error = Bias² + Variance + Noise, so minimizing one component while ignoring the other leads to suboptimal models. The goal is to minimize total error, which requires balancing both terms. With limited training data, accepting higher bias in exchange for lower variance is often the correct engineering decision.

Question 4

As the size of the training dataset grows, model variance generally decreases, even without changing the model architecture.

• T. True
• F. False

Answer: True

Variance is the model's sensitivity to the specific sample of training data. With more data, any single training sample is a more reliable estimate of the underlying data distribution, so the same model architecture trained on different large samples will produce more similar predictions. This is why complex models become viable with abundant data — the variance penalty decreases as more data constrains the model. With very little data, high-variance models are especially dangerous, which is why simpler models often win in low-data regimes.

Question 5

Why does increasing model complexity reduce bias but increase variance? Explain the mechanism in terms of what the model is learning.

Model answer: A simpler model (e.g., a linear function) makes strong assumptions about the form of the relationship between inputs and outputs. These assumptions are usually wrong to some degree — they produce systematic error (bias) that persists no matter how much data you train on. A more complex model (e.g., a high-degree polynomial) makes weaker assumptions and can approximate any shape, so it can more accurately fit the true underlying pattern — reducing bias. However, this flexibility also means the model can fit the noise in the specific training sample, not just the signal. Different training samples have different noise patterns, so the complex model's predictions vary more across samples — high variance. Complexity trades the rigidity of fixed assumptions (bias) for the instability of fitting noise (variance).

The decomposition Error = Bias² + Variance + Noise makes this precise. Bias reflects how wrong the average prediction is; variance reflects how much predictions fluctuate around that average. Adding complexity reduces the average error (bias) while inflating the fluctuation (variance). The optimal complexity minimizes their sum, not either one alone.