Risk of Peeking#
The inflated false-positive risk from repeatedly checking a fixed-horizon test early.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Peeking is looking at an experiment’s results before it officially ends and acting on that interim data. The risk of peeking is the inflated false-positive (Type I) rate that results from repeatedly checking and stopping as soon as significance appears.
Why it breaks the test#
A fixed-horizon test assumes a single look at a predetermined sample size. Each extra peek is another independent chance for noise to cross \(p < 0.05\), so the true error rate compounds far above the nominal \(\alpha\). Peek ten times at a 5% threshold and the real false-positive rate can reach 20–30% — this compounding is called alpha inflation.
What it costs#
Concretely: a button test shows A ahead on day 1 (\(p = 0.04\)), you stop and crown A — but over the full two weeks B would have won. The early stop produced a false conclusion, and at scale that means wrong launches, lost revenue, and eroded trust in experimentation.
How to avoid it#
Four routes: predefine the sample size and duration and only check at the end; use sequential testing / alpha-spending designs built for interim looks (group sequential, O’Brien–Fleming, Pocock); use Bayesian methods designed for continuous monitoring; or, if peeks are unavoidable, apply multiplicity corrections (Bonferroni, Holm). The peeking problem is precisely why the whole machinery of fixed-horizon and sequential testing exists.
Theme: A/B Testing & Experimentation · All terminology
Hint
Mind map — connected ideas
Stopping Rules · Fixed-Horizon Testing · Group Sequential Testing · O’Brien–Fleming (OBF) Method · Pocock Method · Type I Error
Hint
More in A/B Testing & Experimentation
A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect
See also
Source article Adapted (context, re-expressed) in our own words from: Risk of Peeking (insightful-data-lab.com).