Notation#

Part 4 · Stage 14 · 🛡️ Robustness & Missing Data · Lesson 118 of 144 · advanced

◀ Previous · Robust regression using t-distributed errors · Next · Multiple imputation ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

A language for missingness#

Missing data need notation before they need methods, because the right analysis depends entirely on why the data are missing. This lesson sets up the framework — due to Rubin — that makes “why” a precise, model-able quantity rather than a vague worry.

The pieces#

Split the complete data into what you see and what you do not, and add an indicator for which is which:

\[\begin{split}y = (y_{\text{obs}}, y_{\text{mis}}), \qquad R_{ij} = \begin{cases} 1 & y_{ij} \text{ observed} \\ 0 & y_{ij} \text{ missing.} \end{cases}\end{split}\]

The missingness indicator \(R\) is itself data — a matrix you fully observe — and the object that makes the theory work is its distribution \(p(R \mid y, \phi)\), the missingness mechanism. The question is how \(R\) depends on the values \(y\), including the ones you cannot see.

Three mechanisms#

Rubin’s taxonomy, in decreasing order of convenience:

MCAR — missing completely at random. \(p(R \mid y) = p(R)\): missingness is independent of all data, observed and missing alike. Dropped records are then a random subsample, so complete-case analysis is unbiased (if wasteful). Rarely true.
MAR — missing at random. \(p(R \mid y) = p(R \mid y_{\text{obs}})\): missingness depends only on observed values. Income missing more often for the young is MAR if age is recorded. This is the workhorse assumption.
MNAR — missing not at random. Missingness depends on the unobserved values themselves — income missing because it is high. Here the mechanism cannot be ignored and must be modelled explicitly.

Ignorability#

The payoff is a precise condition. When data are MAR and the missingness parameters \(\phi\) are distinct from the model parameters \(\theta\), the mechanism is ignorable: the term \(p(R \mid y_{\text{obs}}, \phi)\) factors out of the likelihood for \(\theta\), so you may model the data and simply ignore \(R\). This is the same ignorability condition met in Stage 7 for data collection — MAR plus parameter distinctness — now applied to missing values. Under it, Bayesian inference proceeds by treating \(y_{\text{mis}}\) as unknown parameters and integrating them out, which is exactly what imputation does.

The catch#

MAR versus MNAR cannot be tested from the data — distinguishing them needs the very values that are missing. The assumption rests on subject knowledge (why would these be missing?) and, where it is doubtful, on a sensitivity analysis across plausible MNAR mechanisms. The notation’s value is exactly this: it turns an untestable worry into an explicit assumption you can state, defend, and vary.

Hint