Calibration quality (Model Calibration)#
How well predicted probabilities match real-world frequencies.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Calibration quality describes how well a model’s predicted probabilities match reality — a well-calibrated model that says “80% confident” is right about 80% of the time. It is a property of the probabilities, separate from whether the model is accurate.
Calibration vs accuracy#
A model can be accurate yet badly calibrated (right often, but its confidence numbers are meaningless) or calibrated yet weakly discriminating. The two are distinct axes, which is why proper scores like log loss and Brier — which reward both — are read together with pure calibration measures.
Why it matters#
Whenever probabilities feed decisions — thresholds, expected-value calculations, downstream systems — calibration is essential, and modern deep networks are typically overconfident. It is measured with ECE, reliability curves, and Brier score, and repaired post-hoc with temperature, Platt, or isotonic scaling.
Theme: Probability Calibration · All terminology
Hint
Mind map — connected ideas
Expected Calibration Error (ECE) · Reliability Curves (also called Calibration Curves) · Temperature Scaling · Overconfident · Confidence Level · Brier Score
Hint
More in Probability Calibration
Adaptive ECE (Expected Calibration Error with Adaptive Binning) · Brier Score · Expected Calibration Error (ECE) · Isotonic Regression · Maximum Calibration Error (MCE) · Murphy’s Decomposition · Overconfident · Platt Scaling · Reliability Curves (also called Calibration Curves) · Temperature Scaling · Underconfident
See also
Source article Adapted (context, re-expressed) in our own words from: Calibration quality (Model Calibration) (insightful-data-lab.com).