📉 Strictly Proper Scoring Rules

Strictly Proper Scoring Rules#

Scoring rules minimised only by reporting the true probability distribution.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A scoring rule assigns a numerical score to a probabilistic forecast given the outcome that materializes. It is proper if the forecaster’s expected score is maximized by reporting the true distribution, and strictly proper if that maximum is unique — attained only at the truth:

\[S(p, q) \;\le\; S(q, q) \quad \text{for all } p, q, \qquad \text{with equality} \iff p = q.\]

Why it matters#

Strict propriety makes honesty optimal: a forecaster cannot improve the expected score by hedging or shading probabilities away from their true beliefs. That single property is why these rules serve both as training objectives (to calibrate probabilistic models) and as evaluation metrics (to rank forecasts fairly). Foundational reference: Gneiting & Raftery (2007).

Common examples#

The logarithmic score, which is also local (it depends only on the density assigned to what actually happened):

\[S(q, x) = \log q(x) \quad\text{(its negative, } -\log q(x)\text{, is the log loss / NLL).}\]

Others include the Brier score for probabilities and the CRPS and pinball (quantile) loss for full distributions and quantiles.

Theme: Risk & Probabilistic Forecasting · All terminology

Hint

Mind map — connected ideas

Probabilistic Forecasts · Probability Forecasts · Continuous Probabilistic Forecasts · Point Forecasts · Quantile Regression · Forecast Error

Hint

More in Risk & Probabilistic Forecasting

Continuous Probabilistic Forecasts · Continuous Ranked Probability Score (CRPS) · Deterministic forecasts · Full Distribution · Pinball Loss (a.k.a. Quantile Loss) · Point Forecasts · Predicting Percentiles · Prediction Intervals (PI) · Probabilistic Forecasts · Probabilistic Scoring · Probability Forecasts · Quantile Forecasts · Quantile Level · Quantile Regression