Bayesian Sequential Testing#
Continuously monitoring an experiment in a Bayesian framework, valid to stop at any time.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Bayesian sequential testing evaluates evidence continuously (or at interim looks) using Bayesian updating, instead of frequentist p-values against a fixed \(\alpha\). At every step you compute a posterior probability or Bayes factor and decide to stop for \(H_1\), stop for \(H_0\), or keep sampling — with no pre-specified sample size.
The two evidence metrics#
Posterior probability — via Bayes’ theorem, \(P(H \mid \text{data}) = P(\text{data} \mid H)\,P(H) / P(\text{data})\).
Bayes factor — \(\text{BF} = P(\text{data} \mid H_1)/P(\text{data} \mid H_0)\); \(\text{BF} > 1\) favours \(H_1\), with \(\text{BF} > 10\) and \(\text{BF} < 1/10\) as strong-evidence thresholds.
Decision rules#
Stop for efficacy — \(P(H_1 \mid \text{data}) > 0.95\) (or \(\text{BF} > 10\)).
Stop for futility — \(P(H_0 \mid \text{data}) > 0.95\) (or \(\text{BF} < 1/10\)).
Continue — evidence still inconclusive.
The key property: continuous monitoring does not inflate the Type I error rate, so peeking is allowed.
Example#
An A/B test with equal priors: after 5,000 visitors the posterior that B beats A is 0.93 → keep sampling; after 8,000 it reaches 0.97 → stop and conclude B is better.
Vs frequentist sequential#
The upside: no fixed horizon (stop anytime), intuitive probability statements (“97% chance B is better”), prior information can be folded in, and error is controlled without α-spending corrections. The downside: it needs a prior (a subjective choice that can sway results) and more computation. Where SPRT and group-sequential methods speak in p-values and likelihood ratios with α-spending, Bayesian sequential testing speaks in posteriors and Bayes factors and treats multiple looks as a non-issue.
Theme: A/B Testing & Experimentation · All terminology
Hint
Mind map — connected ideas
Posterior Probability · Bayesian Stopping Rules · Sequential Settings · Likelihood Ratio (LR) · Frequentist · Conversion Rate Uplift
Hint
More in A/B Testing & Experimentation
A/B Testing · A/B/n Test · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect
See also
Source article Adapted (context, re-expressed) in our own words from: Bayesian Sequential Testing (insightful-data-lab.com).