🧫 Bayesian Stopping Rules

Bayesian Stopping Rules#

Criteria for ending an experiment based on posterior quantities, such as probability of being best.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A Bayesian stopping rule is a predefined criterion — based on posterior probabilities, Bayes factors, or credible intervals — for when to stop collecting data in an experiment. Where a frequentist test fixes the sample size to control the Type I error rate \(\alpha\), a Bayesian rule watches the strength of evidence and stops once it is decisive. In a sentence: stop when the posterior probability of a hypothesis is high enough.

Three kinds of rule#

Posterior-probability threshold — stop and accept \(H_1\) once \(P(H_1 \mid \text{data}) > 0.95\), or stop for futility if \(P(H_0 \mid \text{data}) > 0.95\).
Bayes-factor threshold — stop when the Bayes factor \(\text{BF} = P(\text{data} \mid H_1) / P(\text{data} \mid H_0)\) reaches a strong level on Jeffreys’ scale (\(\text{BF} > 10\) for \(H_1\), \(\text{BF} < 1/10\) for \(H_0\); otherwise keep sampling).
Credible-interval rule — stop when the posterior credible interval for the effect excludes zero (e.g. a 95% interval of \([0.01, 0.03]\) for a conversion-rate difference).

Why the peeking is allowed#

The headline advantage: continuous monitoring does not inflate false positives the way repeated frequentist peeking does. The results are also directly interpretable — “there is a 97% probability that B beats A” — priors can be folded in, and overwhelming evidence lets you stop early and save traffic.

Worked example#

Control converts at 5%; the treatment shows 5.6% after 5,000 users, and the posterior gives \(P(p_B > p_A \mid \text{data}) = 0.97\). With a “stop if posterior > 0.95” rule, the decision is to stop and roll out B.

The catch#

Stopping rules depend on the prior (which can sway the result), are more computationally intensive (often Monte Carlo / MCMC), and ask stakeholders to think in posterior probabilities rather than the p-values they may know better. Versus frequentist fixed-horizon or α-spending designs, the trade is intuitive probability statements and peeking freedom for prior-sensitivity and compute.

Theme: A/B Testing & Experimentation · All terminology

Hint

Mind map — connected ideas

Sequential Settings · Posterior probability of uplift · Bayesian Sequential Testing · Posterior Probability · Frequentist · A/B Testing

Hint

More in A/B Testing & Experimentation

A/B Testing · A/B/n Test · Bayesian Sequential Testing · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect