A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
Ensemble methods combine multiple learners reducing variance, bias, or both. Bagging reduces variance; boosting reduces bias by sequentially correcting errors. Stacking uses meta-learners combining base learners. Diversity among learners is critical for performance.
From your work with decision trees and random forests, you already have intuition for the core insight behind ensembles: a committee of imperfect models can outperform any single model if their errors are sufficiently uncorrelated. A single decision tree is unstable — small changes in training data can produce a completely different tree structure. But if you train many trees on different random subsets of the data and average their predictions, the idiosyncratic errors of individual trees cancel out while the genuine signal reinforces. This is bagging (bootstrap aggregating), and it primarily reduces variance without significantly increasing bias. Random forests extend bagging by also randomizing feature selection at each split, further decorrelating the trees.
Boosting attacks the problem from the opposite direction. Instead of training independent models and averaging them, boosting trains models sequentially, with each new model specifically targeting the mistakes of the previous ensemble. In AdaBoost, misclassified examples receive higher weights so the next learner focuses on the hard cases. In gradient boosting, each new model is fit to the residual errors — the difference between the current ensemble's predictions and the true values. Because each new model corrects systematic errors rather than random noise, boosting primarily reduces bias. The tradeoff is that boosting is more prone to overfitting than bagging, especially with noisy data, because it can learn to fit the noise if run for too many iterations. Learning rate (shrinkage) and early stopping are the standard safeguards.
Stacking (stacked generalization) takes a different approach entirely. Instead of combining base learners through simple averaging or weighted voting, stacking trains a meta-learner that learns the optimal way to combine base model predictions. You might train a decision tree, a logistic regression, and a neural network as base learners, then feed their predictions as features into a second-level model (often a simple linear model) that learns which base learner to trust in which situations. The key requirement is that the meta-learner must be trained on out-of-fold predictions from the base learners — if you use in-sample predictions, the meta-learner will simply learn to trust whichever base model overfits most.
The unifying principle across all ensemble methods is diversity. If all models make the same errors, combining them helps nothing — you just get the same wrong answer more confidently. Bagging creates diversity through data resampling; random forests add feature randomization; boosting creates diversity by reweighting examples; stacking creates diversity by using fundamentally different model families. The theoretical guarantee is precise: for regression with averaging, the ensemble error equals the average error of individual models minus their average pairwise diversity. This means that even mediocre models, if sufficiently diverse, can combine into a strong ensemble — a result that explains why ensemble methods consistently dominate machine learning competitions and production systems.