📉 Probabilistic Scoring

Probabilistic Scoring#

Evaluating forecasts by how well their predicted probabilities match outcomes.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Probabilistic scoring evaluates a probabilistic forecast — a whole predictive distribution — against the outcome that occurs, using a scoring rule (a loss function for distributions). To be trustworthy the rule should be strictly proper, so a forecaster minimizes the expected score only by reporting their true distribution.

The workhorse: CRPS#

The Continuous Ranked Probability Score is the most-used score for real-valued forecasts. It integrates the squared gap between the forecast CDF \(F\) and the step CDF of the observation \(y\), and is negatively oriented (lower is better):

\[\mathrm{CRPS}(F, y) = \int_{-\infty}^{\infty} \big(F(x) - \mathbb{1}\{x \ge y\}\big)^2 \, dx.\]

It generalizes the MAE (for point forecasts) and the Brier score (for binary ones), reducing to them in those cases.

What it captures#

A proper score rewards both calibration (probabilities match reality) and sharpness (tight distributions) — the CRPS in fact decomposes into calibration, discrimination and uncertainty parts. The log score is a local alternative that looks only at the density assigned to the outcome.

Theme: Risk & Probabilistic Forecasting · All terminology

Hint

Mind map — connected ideas

Strictly Proper Scoring Rules · Continuous Probabilistic Forecasts · Probabilistic Forecasts · Full Distribution · Cumulative Distribution Function (CDF) · Quantile Forecasts

Hint

More in Risk & Probabilistic Forecasting

Continuous Probabilistic Forecasts · Continuous Ranked Probability Score (CRPS) · Deterministic forecasts · Full Distribution · Pinball Loss (a.k.a. Quantile Loss) · Point Forecasts · Predicting Percentiles · Prediction Intervals (PI) · Probabilistic Forecasts · Probability Forecasts · Quantile Forecasts · Quantile Level · Quantile Regression · Return Distribution