🌊 Drift Detection

Drift Detection#

Monitoring for changes in data or target distributions that degrade a model.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Drift detection is the practice of monitoring for changes in the data distribution or model behaviour over time that can erode performance. Because real-world data evolves — seasonality, new users, shifting business conditions — a model that was accurate at launch can silently decay, and drift detection is what triggers retraining or recalibration before that decay bites.

The three drifts#

Covariate (feature) drift: the input distribution moves — 80% desktop users become 60% mobile. Label drift (prior shift): the target mix changes — fraud rises from 1% to 3%. Concept drift: the relationship between features and label changes — the same features no longer predict fraud because the tactics evolved. Only concept drift necessarily breaks the learned mapping; the others may or may not.

How it’s detected#

Four families. Statistical tests compare old and new distributions (Kolmogorov–Smirnov, chi-square, PSI — population stability index). Distance measures quantify the gap (KL and Jensen–Shannon divergence, Wasserstein, MMD). Classifier two-sample tests train a model to tell “old” from “new” — if it succeeds, the distributions differ. And performance monitoring tracks loss or AUC over time, though as a lagging signal it only fires once labels arrive.

In practice#

Set baselines from training data, compare rolling windows (last 24h vs last 4 weeks), fire alerts past a threshold (e.g. PSI > 0.2), and wire the signal into the pipeline to retrain or recalibrate. A credit model trained on 2022 data might, by 2025, see income shift toward gig workers — PSI flags the covariate drift and the monitor flags concept drift, triggering a retrain. The payoff is avoiding silent degradation, protecting fairness (drift can hit subgroups unevenly), and meeting compliance demands for monitoring.

Theme: Distribution Shift & Drift · All terminology

Hint

Mind map — connected ideas

Data Drift · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · PSI (Population Stability Index) · Recalibration · Re-scoring

Hint

More in Distribution Shift & Drift

Cardinality in Categorical Data · Categorical Drift · Categorical Explosions · Classifier Two-Sample Tests (C2STs) · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · Data Drift · Dataset Shift · Drift Guardrails · Energy Distance · Jensen–Shannon (JS) Divergence · KS shift (Kolmogorov–Smirnov shift) · Kullback–Leibler (KL) Divergence · Label Drift (a.k.a. Target Drift)