🧫 Group Sequential Testing

Group Sequential Testing#

Frequentist designs allowing interim analyses at planned points with adjusted error spending.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Group sequential testing lets you analyse accumulating data at several pre-planned interim points during an experiment — not just once at the end — and at each look decide to stop for efficacy (the effect is clearly there), stop for futility (it clearly isn’t), or continue. It is standard in clinical trials and increasingly used in A/B testing where stopping early saves resources.

The peeking problem it solves#

Repeatedly checking a fixed-\(\alpha\) test and stopping the moment \(p < 0.05\) inflates the Type I error badly — with many looks, the chance of a false positive can climb to 20–30%. Group sequential designs fix this with an α-spending function that divides the error budget across looks so the overall Type I rate stays at \(\alpha\).

How it works#

Plan the number of interim analyses up front, then use an α-spending rule to set a significance cutoff at each: early looks get stricter thresholds, later looks more lenient ones, and you stop as soon as a threshold is crossed.

The α-spending rules#

O’Brien–Fleming — very strict early, lenient late (final ≈ fixed \(\alpha\)).
Pocock — the same moderate threshold at every look.
Lan–DeMets — a flexible spending function that allocates \(\alpha\) adaptively, without fixing the look times in advance.

Example#

A trial of 1,000 patients, analysed every 250 with total \(\alpha = 0.05\) under an O’Brien–Fleming schedule, might spend \(\alpha = 0.001, 0.01, 0.02, 0.04\) across the four looks. A p-value of 0.008 at 500 patients crosses the second bound → stop early for efficacy.

Where it sits#

It is the middle ground between extremes: more efficient than the fixed-horizon A/B test (one look, may waste data) and statistically valid unlike naive peeking (which inflates false positives), while bandit / adaptive methods go further still by reallocating traffic continuously.

Theme: A/B Testing & Experimentation · All terminology

Hint

Mind map — connected ideas

O’Brien–Fleming (OBF) Method · Pocock Method · Stopping Rules · Sequential Settings · Traditional A/B Test (Fixed-Horizon A/B Test) · Type I Error

Hint

More in A/B Testing & Experimentation

A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect