🌊 Dataset Shift

Dataset Shift#

Any mismatch between training and deployment data distributions.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Dataset shift is when the training data distribution differs from the test / production distribution — formally \(P_{\text{train}}(X, Y) \neq P_{\text{test}}(X, Y)\). Because a model learns its patterns from training data, a shift makes it perform worse in the real world. It is the formal umbrella over the whole drift family.

The three types#

Covariate shift: \(P(X)\) changes but \(P(Y \mid X)\) stays — a spam filter trained on old emails, tested on new ones. Prior (label) shift: \(P(Y)\) changes but \(P(X \mid Y)\) stays — fraud is 1% in training but 5% in production. Concept shift: \(P(Y \mid X)\) itself changes — the meaning of a label evolves, the hardest case to handle.

Detecting it#

Use statistical tests (KS, chi-square, PSI, KL-divergence), a train-versus-test discriminator (if a classifier can tell the two sets apart, they differ), and production monitoring of accuracy, AUC and calibration.

Coping#

Reweight samples by importance, \(w(x) = \frac{P_{\text{test}}(x)}{P_{\text{train}}(x)}\), for covariate shift; resample to match real prevalence; apply domain adaptation; retrain continually; or build robust, invariant models. A model trained on one hospital’s older, less diverse patients loses accuracy on another’s younger, more diverse population.

Theme: Distribution Shift & Drift · All terminology

Hint

Mind map — connected ideas

Covariate Drift (a.k.a. Covariate Shift) · Concept Drift · Data Drift · PSI (Population Stability Index) · KS Statistic (Kolmogorov–Smirnov Statistic) · Label Noise

Hint

More in Distribution Shift & Drift

Cardinality in Categorical Data · Categorical Drift · Categorical Explosions · Classifier Two-Sample Tests (C2STs) · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · Data Drift · Drift Detection · Drift Guardrails · Energy Distance · Jensen–Shannon (JS) Divergence · KS shift (Kolmogorov–Smirnov shift) · Kullback–Leibler (KL) Divergence · Label Drift (a.k.a. Target Drift)