Traditional A/B Test (Fixed-Horizon A/B Test)#
A test analysed once at a pre-committed sample size in order to control error rates.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
A traditional A/B test — equivalently a fixed-horizon A/B test — is the classical approach: predefine a sample size or duration, collect data until that point, and run the hypothesis test exactly once at the end. No interim decisions.
The procedure#
State the hypotheses (\(H_0\): \(CR_A = CR_B\)).
Choose a significance level \(\alpha\) (typically 0.05).
Run an a-priori power analysis to find the required sample size \(n\).
Fix the horizon — e.g. “stop at 10,000 users per variant.”
Collect data to that horizon.
Run the test (a two-proportion z-test for conversion rates).
Reject or fail to reject \(H_0\).
The defining feature is no peeking: because the analysis happens once, at a pre-committed sample size, the Type I error stays at \(\alpha\).
Example#
To test a new button colour with a 5% baseline and a minimum detectable lift of +10%, an a-priori power analysis might call for ~8,000 users per group. You run until each arm hits 8,000, then run a two-proportion z-test, and only then decide whether B beats A.
Strengths and limits#
It is rigorous, widely understood, and easy to explain, and it controls Type I error cleanly when its assumptions hold. The cost is inflexibility: you must wait for the full horizon, which wastes traffic when one variant is clearly better early, and it can’t adapt in real time. The modern alternatives relax exactly this — sequential testing (interim looks via α-spending), Bayesian A/B (probability of superiority, peek freely), and multi-armed bandits (shift traffic to the winner as you learn).
Theme: A/B Testing & Experimentation · All terminology
Hint
Mind map — connected ideas
Type I Error · Stopping Rules · Sequential Settings · Bayesian Sequential Testing · Bandit Algorithms · Conversion Rate Uplift
Hint
More in A/B Testing & Experimentation
A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Treatment Effect
See also
Source article Adapted (context, re-expressed) in our own words from: Traditional A/B Test (Fixed-Horizon A/B Test) (insightful-data-lab.com).