🌊 Kullback–Leibler (KL) Divergence

Kullback–Leibler (KL) Divergence#

An asymmetric measure of how one distribution differs from another.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

KL divergence measures how much one probability distribution P differs from a reference Q — the relative entropy, the expected extra “surprise” from using Q when the truth is P:

\[D_{\text{KL}}(P \,\|\, Q) = \sum_{x} P(x)\,\log\!\frac{P(x)}{Q(x)} \;\ge\; 0,\]

(an integral for continuous distributions). It is 0 only when P = Q.

Its quirks#

KL is asymmetric — \(D_{\text{KL}}(P \| Q) \neq D_{\text{KL}}(Q \| P)\) — so it is not a true distance, and it blows up when Q assigns zero probability where P doesn’t (it needs absolute continuity). It is a directed measure, not a symmetric metric.

Where it’s used#

It is the core of variational methods (the VAE loss), of feature selection, and of drift detection, where it behaves much like PSI — a solid default on large datasets, though its asymmetry means you read it as a degree of drift, not a comparable distance.

Theme: Distribution Shift & Drift · All terminology

Hint

Mind map — connected ideas

Jensen–Shannon (JS) Divergence · Kolmogorov–Smirnov (KS) Test · Data Drift · Statistical Tests · Chi-square (χ²) Test · Cramér’s V

Hint

More in Distribution Shift & Drift

Cardinality in Categorical Data · Categorical Drift · Categorical Explosions · Classifier Two-Sample Tests (C2STs) · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · Data Drift · Dataset Shift · Drift Detection · Drift Guardrails · Energy Distance · Jensen–Shannon (JS) Divergence · KS shift (Kolmogorov–Smirnov shift) · Label Drift (a.k.a. Target Drift)