🏋️ Ensemble

Ensemble#

Combining several models’ predictions to improve accuracy and robustness.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

An ensemble combines multiple models into one stronger predictor. The intuition is the wisdom of the crowd — a group of diverse or individually “weak” models, averaged together, is usually more accurate and more stable than any one of them.

Why use one#

Ensembles reduce variance (steadier, less noise-sensitive predictions), can reduce bias (capturing more complex patterns), and improve robustness (one model’s errors offset by others). They routinely win competitions like Kaggle for exactly these reasons.

The five strategies#

Bagging (bootstrap aggregating) trains models on different random samples and averages or votes — random forest is bagged decision trees. Boosting trains models sequentially, each fixing the last one’s errors (AdaBoost, gradient boosting, XGBoost, LightGBM, CatBoost). Stacking trains base models and a meta-learner to combine them. Voting takes a majority (hard) or averages probabilities (soft). Blending is stacking with the meta-learner trained on a holdout set rather than CV folds.

Trade-offs#

The wins — higher accuracy, resistance to overfitting, complementary patterns from different algorithms — come at a price: more compute, harder interpretability than a single model, and deployment complexity (several models mean more latency and OpEx).

Theme: Model Training & Optimization · All terminology

Hint

Mind map — connected ideas

Re-scoring · Cross-Validation (CV) · Model Distillation (Knowledge Distillation) · Cloud Inference · Bayesian Neural Networks (BNNs) · Model Weights

Hint

More in Model Training & Optimization

Active Learning · Binary Cross-Entropy (BCE) · Deep Ensembles · Early Stopping · Epochs · FLOPs · Full Annotation · Hyperparameter · Label Noise · Log-Odds · Logit Space · Logits · Loss Functions · Model Distillation (Knowledge Distillation)