🧮 Minimum Detectable Lift (MDL)

Minimum Detectable Lift (MDL)#

The smallest effect an experiment is powered to detect reliably.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

The minimum detectable lift (MDL) is the smallest relative change in a metric — conversion, revenue, clicks — that an experiment can reliably detect, given its sample size \(n\), significance level \(\alpha\) and power \(1 - \beta\). It is, in effect, the smallest effect you have decided is worth catching.

Why it matters#

It stops teams over-optimising for trivial effects and forces the design question up front: what improvement is big enough to justify the test? Crucially, the relationship is inverse — the smaller the MDL you want to detect, the larger the sample you need.

The formula#

For conversion rates,

\[\text{MDL} = \frac{p_{\text{treatment}} - p_{\text{control}}}{p_{\text{control}}}.\]

With a 5% baseline, \(\alpha = 0.05\), power 0.80 and 20,000 per variant, the design can detect a 10% relative lift (5% → 5.5%) — so the MDL is +10%. A true lift of only +2% would likely slip past undetected.

MDL vs MDE#

The two are easy to confuse. The minimum detectable effect (MDE) is the smallest absolute change (e.g. +0.5 percentage points); the MDL is the smallest relative change (a percentage lift). For control 5% → treatment 5.5%, the MDE is +0.5 points while the MDL is +10%.

Striking the balance#

Set the MDL too high and you miss small-but-valuable wins; set it too low and the test may need millions of users. The resolution is a negotiation: the business names the smallest improvement worth acting on, and the statistician sizes the experiment to detect at least that.

Theme: Statistical Inference & Power · All terminology

Hint

Mind map — connected ideas

A Priori Power Analysis · Conversion Rate Uplift · Statistical Power · Effect Size (δ) · Two-Proportion Z-Test · Traditional A/B Test (Fixed-Horizon A/B Test)

Hint

More in Statistical Inference & Power

A Priori Power Analysis · Chi-square (χ²) Test · Clopper–Pearson Interval · Compromise Power Analysis · Confidence Intervals (CIs) · Effect Size (δ) · Hypothesis Testing · Kolmogorov–Smirnov (KS) Test · P-Value (probability value) · Post Hoc Power Analysis · Power (1 – β) · Power Analysis · Sample size · Significance Level (α)