🌊 Classifier Two-Sample Tests (C2STs)

Classifier Two-Sample Tests (C2STs)#

Testing for shift by checking whether a classifier can tell two samples apart.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A classifier two-sample test (C2ST) checks whether two datasets come from the same distribution by training a classifier to tell them apart. If the classifier can’t beat chance, the distributions are likely the same; if it can separate them well, they differ — a distribution shift. It is a modern, high-dimensional-friendly alternative to classical tests like the KS test, MMD or energy distance.

How it works#

Three steps. Label dataset A as 0 (say training data) and dataset B as 1 (production). Train a classifier — logistic regression, random forest, neural net — to predict which set a sample came from. Evaluate: accuracy near 50% means the two are indistinguishable (same distribution), while accuracy well above 50% (or a high AUC) signals they differ.

The hypothesis test#

Formally it tests \(H_0: P = Q\) against \(H_1: P \neq Q\), using classifier accuracy or AUC as the test statistic and permutation testing or bootstrapping to get a p-value. Its strengths are working in very high dimensions (where KS or chi-square fail) and using off-the-shelf classifiers; its costs are the compute of training, sensitivity to classifier choice, and the need for enough data.

Where it’s used#

C2STs power data-drift detection (train vs production), generative-model evaluation (real vs generated), and bias detection (comparing subgroups). Concretely: with 10,000 samples from 2024 and 10,000 from 2025, an XGBoost separator reaching AUC 0.90 means the two are easily distinguishable — strong drift.

Theme: Distribution Shift & Drift · All terminology

Hint

Mind map — connected ideas

Representation Shift · Drift Detection · Energy Distance · Data Drift · PSI (Population Stability Index) · Maximum Mean Discrepancy (MMD)

Hint

More in Distribution Shift & Drift

Cardinality in Categorical Data · Categorical Drift · Categorical Explosions · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · Data Drift · Dataset Shift · Drift Detection · Drift Guardrails · Energy Distance · Jensen–Shannon (JS) Divergence · KS shift (Kolmogorov–Smirnov shift) · Kullback–Leibler (KL) Divergence · Label Drift (a.k.a. Target Drift)