🔁 Posterior probability of uplift

Posterior probability of uplift#

The posterior probability that a treatment’s effect exceeds zero (or a chosen threshold).

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The question it answers#

In an experiment with a binary outcome (say conversion), the uplift is the difference in success rates between treatment and control,

\[u = p_T - p_C,\]

and the decision-relevant quantity is not a p-value but the posterior probability that the uplift is positive (or beats a business threshold \(\tau\)):

\[\Pr(u > 0 \mid \text{data}) \quad\text{or}\quad \Pr(u > \tau \mid \text{data}).\]

Beta–Binomial model (binary outcomes)#

With \(s_T\) successes in \(n_T\) trials (treatment) and \(s_C\) in \(n_C\) (control), conjugate Beta priors give Beta posteriors:

\[p_T \mid \text{data} \sim \text{Beta}(\alpha_T + s_T,\; \beta_T + n_T - s_T), \qquad p_C \mid \text{data} \sim \text{Beta}(\alpha_C + s_C,\; \beta_C + n_C - s_C).\]

The difference \(u = p_T - p_C\) has no closed-form CDF, so estimate it by Monte Carlo: draw \(p_T^{(m)}\) and \(p_C^{(m)}\) from their posteriors, form \(u^{(m)} = p_T^{(m)} - p_C^{(m)}\), and approximate

\[\Pr(u > \tau \mid \text{data}) \approx \frac{1}{M}\sum_{m=1}^M \mathbf{1}\{u^{(m)} > \tau\}.\]

The empirical quantiles of \(\{u^{(m)}\}\) give a credible interval for the uplift.

import numpy as np
pT = np.random.beta(1 + sT, 1 + nT - sT, size=200_000)
pC = np.random.beta(1 + sC, 1 + nC - sC, size=200_000)
u  = pT - pC
prob_uplift = np.mean(u > tau)          # Pr(u > tau | data)
ci = np.quantile(u, [0.025, 0.975])     # 95% credible interval

Continuous outcomes (revenue)#

For approximately Normal outcomes with unknown means, a Normal–Inverse-Gamma posterior makes the mean difference \(d = \mu_T - \mu_C\) (approximately) Normal, so the uplift probability is closed-form:

\[\Pr(d > \tau \mid \text{data}) = 1 - \Phi\!\left(\frac{\tau - \hat{d}}{\text{SE}_{\text{post}}}\right).\]

Bayesian logistic / hierarchical regression#

With covariates and a treatment indicator,

\[\Pr(Y=1 \mid x, T) = \text{logit}^{-1}\!\big(\beta_0 + \beta^\top x + \gamma T + \delta^\top (x \cdot T)\big),\]

per-draw segment uplift is computed from posterior coefficient samples, and hierarchical priors borrow strength across small segments to reduce false positives.

What to report, and how to decide#

Decision-friendly outputs: the posterior probability \(\Pr(u > \tau)\), the expected uplift \(\mathbb{E}[u \mid \text{data}]\), a 95% credible interval, and a risk-aware rule — “ship if \(\Pr(u > \tau) \ge q\)” (e.g. \(q = 0.9\)). Practical guidance: use weakly informative priors (Beta(1,1) or Beta(0.5,0.5)) for small samples, prefer \(\Pr(u > \tau)\) over point estimates when a bad launch is costly, model many arms hierarchically, and — unlike fixed-horizon tests — Bayesian posteriors support always-valid monitoring without naive peeking penalties.

Theme: Bayesian Inference · All terminology

Hint

Mind map — connected ideas

A/B Testing · Conversion Rate Uplift · Bayesian Sequential Testing · Posterior · Conversion Rate (CR) · Incremental Conversions

Hint

More in Bayesian Inference

Bayes’ Theorem · Bayesian Correction · Bayesian Decision Theory (BDT) · Bayesian Inference. · Bayesian Neural Networks (BNNs) · Binomial Likelihood · Gaussian Processes (GPs) · Marginal Likelihood (also called The Model Evidence or Integrated Likelihood) · MCMC (Markov Chain Monte Carlo) · Parameter(s) of Interest · Posterior · Posterior belief · Posterior Probability · Prior Belief (or Prior Probability)