Fixed-Horizon Testing#
Testing where the sample size is fixed in advance and analysed only at the end.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Fixed-horizon testing is the general statistical principle behind the classic experiment: decide the sample size \(n\) (or end time) in advance, collect data to that point, and run the hypothesis test exactly once. The “horizon” is that pre-committed stopping point. Applied to an A/B test specifically, this is the traditional A/B test.
Why the horizon matters#
Fixing it up front is what keeps the statistics honest: the Type I error stays at \(\alpha\), peeking is ruled out (checking early and stopping on significance inflates false positives), and the resulting p-values and confidence intervals remain valid under their assumptions.
The procedure#
State \(H_0\) and \(H_1\); choose \(\alpha\) (say 0.05); run an a-priori power analysis to size the sample; fix the horizon; collect to it; test once; decide.
Example#
To detect a +10% lift (5% → 5.5%) at \(\alpha = 0.05\) and power 0.80, an a-priori power analysis calls for ≈ 7,850 users per variant. You stop at that horizon and run a single two-proportion z-test: \(p \le 0.05\) rejects \(H_0\), otherwise you fail to reject.
Strengths and limits#
It is simple, widely accepted, and preserves error guarantees. The price is rigidity: no early stop even when the result is already obvious, wasted samples when an effect is large, and no continuous monitoring. Sequential and adaptive methods (α-spending, Bayesian updating, bandits) trade some of that simplicity for the ability to stop early.
Theme: A/B Testing & Experimentation · All terminology
Hint
Mind map — connected ideas
Traditional A/B Test (Fixed-Horizon A/B Test) · Stopping Rules · Sequential Settings · Type I Error · Bayesian Sequential Testing · Standard Error (SE)
Hint
More in A/B Testing & Experimentation
A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect
See also
Source article Adapted (context, re-expressed) in our own words from: Fixed-Horizon Testing (insightful-data-lab.com).