🧫 True Conversion Rate

True Conversion Rate#

The unknown underlying probability that a user converts, estimated from observed conversions.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

The true conversion rate \(p\) is the actual probability that a user in the whole population converts (clicks, buys, signs up). It is a population parameter — fixed but unknown. What an experiment actually measures is the sample conversion rate \(\hat{p}\), an estimate of \(p\).

Parameter vs estimate#

\[p = \frac{\text{conversions in the population}}{\text{users in the population}}, \qquad \hat{p} = \frac{x}{n},\]

where \(x\) is conversions in the sample and \(n\) the sample size. We can rarely see the whole population, so we work with \(\hat{p}\) and quantify its uncertainty.

Example#

1,000 users see version A and 50 convert, so \(\hat{p}_A = 50/1000 = 0.05\) (5%). The true rate might be \(p = 0.052\), but we never observe it exactly — only estimate it.

Confidence interval for p#

Because \(\hat{p}\) carries sampling error, a Wald confidence interval brackets the likely range of \(p\):

\[\text{CI} = \hat{p} \pm z_{\alpha/2}\, \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}.\]

With \(\hat{p} = 0.05, n = 1000\) and 95% confidence, the standard error is \(\sqrt{0.05 \times 0.95 / 1000} \approx 0.0069\), giving \(0.05 \pm 1.96 \times 0.0069 \approx [0.036, 0.064]\) — we’re 95% confident the true rate lies between 3.6% and 6.4%.

Why it matters#

In A/B testing we never know either group’s true rate; we estimate both with \(\hat{p}\) and use a two-proportion z-test to judge whether the observed difference is real evidence of a difference in the true conversion rates — the actual quantity of interest.

Theme: A/B Testing & Experimentation · All terminology

Hint

Mind map — connected ideas

Standard Error (SE) · Conversion Rate (CR) · Parameter(s) of Interest · Conversion Rate Uplift · Frequentist · A/B Testing

Hint

More in A/B Testing & Experimentation

A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test)