Adaptive ECE (Expected Calibration Error with Adaptive Binning)#
A calibration error using adaptive bins so each holds a similar count.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Adaptive ECE measures a classifier’s miscalibration — the gap between its confidence and its actual accuracy — using equal-count bins. Like standard Expected Calibration Error, it is a weighted average of |accuracy − confidence| across bins:
The “adaptive” part changes only how the bins are drawn.
The problem it fixes#
Standard ECE uses fixed equal-width bins ([0.0–0.1], …), so when predictions cluster (modern nets pile probabilities near 1.0), some bins hold few or zero samples and give noisy estimates. Adaptive binning instead makes each bin hold the same number of predictions, so bin widths vary with the data.
Why it helps#
Equal-count bins yield a more stable, fairer calibration estimate on skewed predictions, where equal-width ECE is unreliable. It shares ECE’s caveat, though: the result still depends on the number of bins, and neither is a proper scoring rule.
Theme: Probability Calibration · All terminology
Hint
Mind map — connected ideas
Maximum Calibration Error (MCE) · Murphy’s Decomposition · Confidence Level · Temperature Scaling · Overconfident · Underconfident
Hint
More in Probability Calibration
Brier Score · Calibration quality (Model Calibration) · Expected Calibration Error (ECE) · Isotonic Regression · Maximum Calibration Error (MCE) · Murphy’s Decomposition · Overconfident · Platt Scaling · Reliability Curves (also called Calibration Curves) · Temperature Scaling · Underconfident
See also
Source article Adapted (context, re-expressed) in our own words from: Adaptive ECE (Expected Calibration Error with Adaptive Binning) (insightful-data-lab.com).