Designed experiments#

Part 2 · Stage 7 · 🗳️ Data Collection & Decisions · Lesson 053 of 144 · intermediate

◀ Previous · Sample surveys · Next · Sensitivity and the role of randomization ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Choosing the mechanism yourself#

In a survey the inclusion mechanism is something you must discover. In an experiment you choose it. That is the whole advantage: the assignment mechanism \(p(W \mid y, \phi)\) — which units get treatment, \(W_i \in \{0, 1\}\) — is known, controlled, and probabilistic by construction, so ignorability is guaranteed rather than assumed.

Causal inference as missing data#

Each unit has two potential outcomes: \(y_i(1)\) under treatment and \(y_i(0)\) under control. You observe exactly one. The causal effect \(y_i(1) - y_i(0)\) is therefore unobservable for every individual — one of its two terms is always missing. Causal inference is a missing-data problem, and the assignment mechanism is the missingness mechanism:

\[\Pr\bigl(W \mid y(0), y(1), X, \phi\bigr) .\]

Complete randomisation makes this depend on nothing (the parallel of MCAR). Stratified or blocked randomisation makes it depend only on covariates \(X\) (the parallel of MAR, ignorable given \(X\)). Either way the mechanism factors out — provided the design variables are in the model.

The Bayesian analysis#

Once ignorability holds, fitting is unremarkable: model the outcome, include the design, and read the causal effect off the posterior. Blocking becomes a hierarchical term; the estimand is a function of draws.

import pymc as pm
with pm.Model():
    a_block = pm.Normal("a_block", 0, pm.HalfNormal("s", 1), shape=n_blocks)  # blocking
    tau  = pm.Normal("tau", 0, 1)                    # treatment effect
    mu   = a_block[block] + tau * W
    pm.Normal("y", mu, pm.HalfNormal("sigma", 1), observed=y)
    idata = pm.sample()
# P(effect > 0 | data) is a count over draws; the design justified the model

Design earns you the likelihood#

Two consequences worth stating. Blocking on a covariate must be matched by including it in the model: randomising within blocks and then ignoring blocks discards the design’s benefit and can mis-state uncertainty. And randomisation buys ignorability, not precision — a badly designed randomised experiment gives an unbiased answer with an interval too wide to act on. The next lesson asks exactly what randomisation does and does not protect against.

Hint