🌊 Maximum Mean Discrepancy (MMD)

Maximum Mean Discrepancy (MMD)#

A kernel-based distance between distributions for two-sample and drift tests.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Maximum mean discrepancy (MMD) is a statistical test for the difference between two distributions \(P\) and \(Q\) from samples. Under a chosen kernel, MMD is 0 exactly when the distributions match, and grows as they diverge. It is widely used to detect drift, compare train versus test data, and evaluate generative models (GANs, VAEs, diffusion).

The idea#

MMD compares the mean embeddings of the two distributions in a reproducing kernel Hilbert space (RKHS):

\[\text{MMD}^2(P, Q) = \left\| \mu_P - \mu_Q \right\|_{\mathcal{H}}^2,\]

where \(\mu_P = \mathbb{E}_{x \sim P}[\phi(x)]\) is the mean embedding of \(P\) under the feature map \(\phi\) of a kernel \(k(x, y)\).

Estimating it#

The kernel trick gives an estimate from samples without ever touching the infinite-dimensional space — averaging within-\(P\) kernels plus within-\(Q\) kernels minus twice the cross kernels:

\[\widehat{\text{MMD}}^2 = \frac{1}{m^2}\sum_{i,i'} k(x_i, x_{i'}) + \frac{1}{n^2}\sum_{j,j'} k(y_j, y_{j'}) - \frac{2}{mn}\sum_{i,j} k(x_i, y_j),\]

usually with a Gaussian RBF kernel.

Where it’s used#

A small MMD means similar samples, a large one means different. It powers drift detection (train vs production, covariate drift), GAN evaluation (generated vs real), and domain adaptation (aligning source and target features, as in MMD-regularised networks). Train a fraud model on last year’s data \(P\); if this year’s \(Q\) gives a high MMD, you have covariate drift and likely need to retrain.

Theme: Distribution Shift & Drift · All terminology

Hint

Mind map — connected ideas

Energy Distance · Classifier Two-Sample Tests (C2STs) · Drift Detection · Covariate Drift (a.k.a. Covariate Shift) · Representation Shift · PSI (Population Stability Index)

Hint

More in Distribution Shift & Drift

Cardinality in Categorical Data · Categorical Drift · Categorical Explosions · Classifier Two-Sample Tests (C2STs) · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · Data Drift · Dataset Shift · Drift Detection · Drift Guardrails · Energy Distance · Jensen–Shannon (JS) Divergence · KS shift (Kolmogorov–Smirnov shift) · Kullback–Leibler (KL) Divergence