🧫 Treatment Effect

Treatment Effect#

The causal difference in outcome between treated and untreated units.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

The treatment effect is the causal impact of an intervention versus a control — how much did the treatment change the outcome compared with what would have happened without it? In the potential-outcomes framework,

\[\text{Treatment Effect} = Y(1) - Y(0),\]

where \(Y(1)\) is the outcome if the unit is treated and \(Y(0)\) the outcome if it is not.

The fundamental problem#

For any single unit you only ever observe one of \(Y(1)\) or \(Y(0)\) — never both — so the individual effect is unobservable. This fundamental problem of causal inference is why we estimate averages instead of individual effects.

The hierarchy of effects#

ITE (individual) — \(Y_i(1) - Y_i(0)\) for one unit; unobservable.
ATE (average) — \(\text{ATE} = \mathbb{E}[Y(1) - Y(0)]\), the population-average effect, the usual target of RCTs and A/B tests.
CATE (conditional average) — \(\text{CATE}(x) = \mathbb{E}[Y(1) - Y(0) \mid X = x]\), the effect within a subgroup defined by covariates \(x\) (age, segment) — the basis of personalised interventions.
LATE (local average) — the effect for a specific compliant subgroup, typically via instrumental variables.

Examples#

A drug with 60% recovery vs 50% on placebo has \(\text{ATE} = +10\) points. A website variant at 5.5% vs 5% conversion has \(\text{ATE} = +0.5\) points, a \((0.055 - 0.05)/0.05 = +10\%\) relative lift.

How it’s estimated#

In a randomised controlled trial (or A/B test), randomisation makes the groups comparable, so the difference in group means is an unbiased estimate of the ATE. In observational data, confounding must be removed with causal-inference tools — matching, regression adjustment, instrumental variables, difference-in-differences or propensity scores. ML methods increasingly estimate heterogeneous (CATE) effects for targeting.

Theme: A/B Testing & Experimentation · All terminology

Hint

Mind map — connected ideas

A/B Testing · Conversion Rate Uplift · Parameter(s) of Interest · Causal Inference · Posterior probability of uplift · Online Experimentation Platforms

Hint

More in A/B Testing & Experimentation

A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Online Experimentation Platforms · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test)