🧫 Stopping Rules

Stopping Rules#

Predefined conditions that determine when to stop collecting data in a test.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A stopping rule is the pre-specified condition for ending an experiment — it says when to stop collecting data and decide. Without one, you can “peek” until the result looks significant, which inflates the Type I error (false-positive) rate. The choice of stopping rule is what makes continuous monitoring valid or invalid.

The families#

Fixed-horizon (the traditional A/B test) — fix the sample size up front (say 10,000 per arm), analyse once at the end. Simple and protects \(\alpha\), but inflexible.
Group-sequential — pre-plan interim looks (e.g. every 25%) and spend the error budget with an α-spending rule (O’Brien–Fleming, Pocock), allowing early stops for efficacy or futility. Efficient; standard in clinical trials.
Sequential probability ratio test (SPRT) — check a likelihood ratio after each observation and stop once it crosses a bound for \(H_1\) or \(H_0\); often uses far fewer samples.
Bayesian — stop when a posterior probability crosses a threshold (e.g. \(P(H_1 \mid \text{data}) > 0.95\)); intuitive and needs no α-spending.
Ethical / practical — in trials, stop for harm, for clear benefit (it’s unethical to withhold), or for futility.

Examples#

An A/B test might run to a fixed 50,000 visitors per arm; or check every 5,000 and stop early if \(p < 0.001\) under an O’Brien–Fleming bound; or stop once the posterior probability that B beats A exceeds 0.95. A clinical trial planned for 1,000 patients might look at 250, 500 and 750 — stopping early for strong benefit, immediately for harm, or for futility.

Why they matter#

Stopping rules prevent p-hacking (stopping the moment things look good), keep Type I error controlled, save time and cost by ending early when results are clear, and protect participants in clinical settings. The dividing line: fixed horizon (one look) versus sequential / adaptive (group-sequential, SPRT, Bayesian) that permit interim looks without inflating false positives.

Theme: A/B Testing & Experimentation · All terminology

Hint

Mind map — connected ideas

Bayesian Stopping Rules · Sequential Settings · Traditional A/B Test (Fixed-Horizon A/B Test) · Frequentist · A/B Testing · Online Experimentation Platforms

Hint

More in A/B Testing & Experimentation

A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect