🎯 Expected Calibration Error (ECE)

Expected Calibration Error (ECE)#

The average gap between confidence and accuracy across probability bins.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Expected Calibration Error summarizes miscalibration in one number — the weighted average gap between a model’s confidence and its accuracy, taken over bins of predictions:

\[\text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{N}\,\big|\mathrm{acc}(B_m) - \mathrm{conf}(B_m)\big|.\]

Geometrically, it is the average distance of the calibration curve from the diagonal.

How it’s computed#

Predictions are grouped into bins by confidence; in each bin you compare the fraction correct (accuracy) to the average confidence, and weight each bin’s gap by its size. The result is bounded in [0, 1] and easy to report — the standard scalar for comparing calibration.

Its caveats#

ECE is bin-dependent (the number and placement of bins move the value) and, being an average, it can hide a badly miscalibrated region behind well-behaved bins. It is also not a proper scoring rule — a trivial model can score low — so it is read with reliability curves and Brier score. Its variants are MCE and Adaptive ECE.

Theme: Probability Calibration · All terminology

Hint

Mind map — connected ideas

Adaptive ECE (Expected Calibration Error with Adaptive Binning) · Maximum Calibration Error (MCE) · Reliability Curves (also called Calibration Curves) · Confidence Level · Temperature Scaling · Brier Score

Hint

More in Probability Calibration

Adaptive ECE (Expected Calibration Error with Adaptive Binning) · Brier Score · Calibration quality (Model Calibration) · Isotonic Regression · Maximum Calibration Error (MCE) · Murphy’s Decomposition · Overconfident · Platt Scaling · Reliability Curves (also called Calibration Curves) · Temperature Scaling · Underconfident