🧫 A/B Testing

A/B Testing#

A randomised experiment comparing two variants to measure an effect.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

An A/B test is a controlled experiment that randomly assigns users to two variants — A (control) and B (treatment) — and measures which performs better on a chosen metric (conversion rate, time on page, retention). Randomization is what lets you read the difference as causal.

How it’s run#

You fix the metric, use a power analysis to set the sample size, pick a statistical test (t-test, chi-square), and choose a significance level α. When the data are in, the test decides whether B’s effect is real or noise.

Its discipline#

The classic A/B test is fixed-horizon — you must wait for the pre-planned sample before deciding. Peeking early and stopping when it looks significant inflates false positives, which is exactly the failure that sequential methods are designed to fix.

Theme: A/B Testing & Experimentation · All terminology

Hint

Mind map — connected ideas

Sequential Testing (also called sequential analysis) · Interleaving Tests · Traditional A/B Test (Fixed-Horizon A/B Test) · Power Analysis · Statistical Tests · Conversion Rate (CR)

Hint

More in A/B Testing & Experimentation

A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect