🎯 Brier Score

Brier Score#

The mean squared error of probabilistic predictions.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

The Brier score is the mean squared error of probabilistic predictions — the average squared gap between the predicted probability and the actual (0/1) outcome:

\[\text{BS} = \frac{1}{N}\sum_{i=1}^{N} (p_i - y_i)^2.\]

Lower is better, with 0 perfect. It is a single sample-level number for binary or multiclass problems.

Why it’s special#

Unlike ECE, the Brier score is a strictly proper scoring rule — it is minimized only by honest probabilities, and by Murphy’s decomposition it splits into calibration plus refinement terms. So a low Brier score means the model is both well-calibrated and discriminating.

Its limitation#

Because it blends calibration and discrimination, the Brier score can’t tell you which is lacking — a sharp-but-miscalibrated model and a calibrated-but-vague one can score similarly. That is why it is reported alongside ECE and reliability curves, which isolate the calibration piece.

Theme: Probability Calibration · All terminology

Hint

Mind map — connected ideas

Expected Calibration Error (ECE) · Murphy’s Decomposition · Strictly Proper Scoring Rules · Continuous Ranked Probability Score (CRPS) · Reliability Curves (also called Calibration Curves) · Log Loss (also called Logarithmic Loss or Cross-Entropy Loss)

Hint

More in Probability Calibration

Adaptive ECE (Expected Calibration Error with Adaptive Binning) · Calibration quality (Model Calibration) · Expected Calibration Error (ECE) · Isotonic Regression · Maximum Calibration Error (MCE) · Murphy’s Decomposition · Overconfident · Platt Scaling · Reliability Curves (also called Calibration Curves) · Temperature Scaling · Underconfident