Sequential Testing (also called sequential analysis)#
Analysing results as data arrive while controlling error from repeated looks.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Sequential testing (sequential analysis) monitors an experiment continuously and lets you stop as soon as the evidence is conclusive — rather than waiting for a fixed sample size. It is built for the streaming data of modern experimentation platforms.
The problem it solves#
Repeatedly checking a fixed-horizon test and stopping when it looks good — peeking — badly inflates the Type I error. Sequential methods like the sequential probability ratio test (SPRT) and group-sequential designs keep the false-positive rate controlled at any time, so early stopping is valid.
The payoff#
Because it can end as soon as a winner (or a dead end) is clear, sequential testing cuts the average sample size and deployment time, exposing fewer users to an inferior variant — at the price of slightly more conservative thresholds to preserve error control.
Theme: A/B Testing & Experimentation · All terminology
Hint
Mind map — connected ideas
A/B Testing · Interleaving Tests · Statistical Tests · Power Analysis · Bootstrap · Confidence Intervals (CIs)
Hint
More in A/B Testing & Experimentation
A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect
See also
Source article Adapted (context, re-expressed) in our own words from: Sequential Testing (also called sequential analysis) (insightful-data-lab.com).