Regularization reduces overfitting by penalizing model complexity. L1 (Lasso) encourages sparsity; L2 (Ridge) shrinks weights. Early stopping halts at validation peak. Dropout randomly removes neurons; batch normalization stabilizes activations. Data augmentation increases effective samples.
From the bias-variance tradeoff, you know that a model with too much capacity memorizes training noise rather than learning the true underlying pattern — it has low bias but high variance, and it generalizes poorly. Regularization is the family of techniques that constrains a model's effective complexity, pushing it toward simpler solutions that generalize better. The core intuition is that you are willing to accept a small increase in training error if it buys a large decrease in test error.
The most classical approach adds a penalty term to the loss function based on the magnitude of the model's weights. L2 regularization (Ridge) adds λ·Σwᵢ², which penalizes large weights quadratically. This doesn't force weights to zero — it shrinks them all toward zero proportionally, producing models that spread influence across many features rather than relying heavily on a few. L1 regularization (Lasso) adds λ·Σ|wᵢ|, which penalizes the absolute values of weights. The geometry of the L1 penalty (a diamond-shaped constraint region) means that optimal solutions often land exactly at zero for some weights, producing sparse models that effectively perform feature selection. If you have studied constrained optimization, you can see both penalties as Lagrangian relaxations of constraints on the weight vector's norm.
Beyond explicit penalties, several techniques regularize through the training *process* rather than the loss function. Early stopping monitors validation loss during training and halts when it begins to rise — the model has not yet had enough iterations to overfit. Dropout randomly deactivates a fraction of neurons during each training step, forcing the network to learn redundant representations that are robust to missing features. At test time, all neurons are active but weights are scaled down to compensate. The effect is similar to training an implicit ensemble of sub-networks. Batch normalization normalizes activations within each mini-batch, which stabilizes gradients and has an incidental regularizing effect by introducing noise through the batch statistics.
Data augmentation takes a different angle entirely: instead of constraining the model, it expands the effective size of the training set. For images, this means applying random flips, rotations, crops, and color jitter to create synthetic training examples that encode known invariances. The model sees more diversity without requiring more real data, which directly reduces overfitting. In practice, strong results come from combining several regularization strategies — for example, L2 penalty plus dropout plus data augmentation — with the strength of each tuned on a validation set. The regularization hyperparameter λ controls the bias-variance tradeoff: too little regularization and the model overfits, too much and it underfits.