🏋️ Deep Ensembles

Deep Ensembles#

Averaging several independently trained networks for accuracy and uncertainty.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A deep ensemble trains several neural networks independently and aggregates their predictions, improving both accuracy and uncertainty estimation. Each member shares the architecture but starts from a different random initialisation (and data order), so the networks settle into different modes of the loss landscape; averaging their outputs cancels errors and reduces variance:

\[\bar{p}(y \mid x) = \frac{1}{M} \sum_{m=1}^{M} p_{\theta_m}(y \mid x).\]

Why it works, and uncertainty#

Because independently-initialised networks explore different functions, their disagreement is informative. Deep ensembles decompose predictive uncertainty into aleatoric (data noise) and epistemic (model) components and produce well-calibrated predictive intervals — rivalling Bayesian neural networks while being far simpler to implement.

Calibration and cost#

They usually still need calibration (for example temperature scaling), especially under distribution shift. The main drawback is that cost grows linearly with the number of members \(M\), which motivates efficient variants such as BatchEnsemble, snapshot ensembles, and spreading members over time.

Where it’s used#

Deep ensembles shine wherever reliable confidence matters as much as the point prediction — climate downscaling, robotic perception and safe human-robot interaction, and low-data transfer learning.

Theme: Model Training & Optimization · All terminology

Hint

Mind map — connected ideas

Decision Trees · Post-hoc Explainability · Bayesian Neural Networks (BNNs) · Model Stability · Uplift Random Forests · SHAP (SHapley Additive exPlanations)

Hint

More in Model Training & Optimization

Active Learning · Binary Cross-Entropy (BCE) · Early Stopping · Ensemble · Epochs · FLOPs · Full Annotation · Hyperparameter · Label Noise · Log-Odds · Logit Space · Logits · Loss Functions · Model Distillation (Knowledge Distillation)