🌊 Off-Distribution

Off-Distribution#

Inputs that fall outside the distribution a model was trained on.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Off-distribution data are points that differ significantly from the distribution the model was trained on — inputs outside its “familiar range”. Models assume production data is drawn from the same distribution as training (the i.i.d. assumption); when that breaks, the input is off-distribution, also called out-of-distribution (OOD).

Examples#

A cats-versus-dogs classifier shown a giraffe is off-distribution. A credit model trained on 2015-2020 applications meets post-COVID borrower behaviour it never saw. A diagnostic model trained on adult MRIs is handed a child’s scan. In each case the input falls outside the learned scope.

Why it’s a problem, and detecting it#

Models optimised for in-distribution data make unreliable or overconfident predictions on OOD inputs, with fairness risks for unseen subgroups. Detection draws on distance metrics (KL, Jensen-Shannon, KS), embedding methods (Mahalanobis or cosine distance in latent space), uncertainty estimation (Bayesian neural nets, deep ensembles, MC dropout), and dedicated OOD classifiers.

Handling it#

Five responses: augment the training data to broaden coverage; adapt the model to the new domain; build robust models with regularisation or adversarial training; add a reject option so the model can abstain (“I don’t know”); and run monitoring pipelines to flag drift in production. Off-distribution is the abrupt cousin of gradual data and concept drift.

Theme: Distribution Shift & Drift · All terminology

Hint

Mind map — connected ideas

IID (Independent and Identically Distributed) · Data Drift · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · Drift Detection · Representation Shift

Hint

More in Distribution Shift & Drift

Cardinality in Categorical Data · Categorical Drift · Categorical Explosions · Classifier Two-Sample Tests (C2STs) · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · Data Drift · Dataset Shift · Drift Detection · Drift Guardrails · Energy Distance · Jensen–Shannon (JS) Divergence · KS shift (Kolmogorov–Smirnov shift) · Kullback–Leibler (KL) Divergence