Bagging, from Breiman's 1996 paper, fits the same model to many bootstrap resamples of the training data and averages the predictions. The reason it works is a variance identity worth memorising. If B estimators each have variance σ² and pairwise correlation ρ, the variance of their average is
rho * sigma^2 + (1 - rho) * sigma^2 / BWhy does bagging reduce variance?
The second term vanishes as B grows. The first does not. Averaging 100 trees with pairwise correlation 0.60 gives a variance of 0.6040 against a single tree's 1.0000, and pushing B to infinity would only reach 0.6000. Decorrelate the trees down to ρ = 0.25 and the same 100 trees give 0.2575, which is 2.35 times better. The floor is what matters, and no quantity of extra trees beats ρσ², which is why decorrelating the members is the only real lever.
A random forest, from Breiman's 2001 paper, is bagging plus exactly that lever: at every split each tree may consider only a random subset of the features. Forcing trees to disagree lowers ρ, and the identity above turns that into lower variance for the ensemble. Each individual tree is worse than it would have been with all features available, and the ensemble is better. Out-of-bag observations, the roughly 37% of rows a given bootstrap sample omits, provide a free held-out estimate.
Advertisement
How does boosting work?
Boosting works the other way. Instead of fitting many independent models and averaging, it fits models in sequence, each one trained on what the previous ones got wrong, and adds them with small weights. Gradient boosting is the dominant form on structured financial data. Boosting typically beats bagging on accuracy and gives up two things in return: it is far more sensitive to noisy labels, because it keeps re-weighting toward the hardest observations, and it can overfit as the number of rounds grows, so the round count is itself a cross-validated parameter.
The honest summary of all of it is that an ensemble's error falls below its average member's because the members' errors partly cancel, and the cancellation is only as good as the members' independence. An ensemble of a hundred nearly identical models is a slow way to fit one model.