General Notation for Statistical Inference#

Part 1 · Stage 1 · 🎲 The Bayesian Idea · Lesson 002 of 144 · beginner

◀ Previous · The three steps of Bayesian data analysis · Next · Bayesian Inference ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The three symbols#

Bayesian writing is remarkably compact once its notation is fixed. Three symbols carry most of the weight:

\(\theta\) — the unobservable quantities: parameters, or anything else you want to learn.
\(y\) — the observed data.
\(\tilde{y}\) — potentially observable data: future or replicated observations you have not yet seen. (Predictors, when present, are written \(x\).)

The joint model#

A full probability model is a joint distribution over the unknowns and the observables, which always factors into prior times likelihood:

\[p(\theta, y) = p(\theta)\, p(y \mid \theta).\]

Everything else follows by conditioning and marginalising. Inference about parameters is \(p(\theta \mid y)\); prediction of new data is the posterior predictive \(p(\tilde{y} \mid y)\), obtained by averaging the likelihood over the posterior.

Exchangeability#

Why is it ever legitimate to model observations as identically distributed? The Bayesian answer is exchangeability: if the labels on the observations carry no information — if any reordering of \(y_1, \dots, y_n\) is equally plausible — then their joint distribution can be written as independent draws given some parameter, mixed over a distribution for that parameter. Exchangeability, not an assumption of “random sampling”, is what licenses the usual iid likelihood, and it becomes the foundation of hierarchical models later in this course.

Read the conditioning bar#

One habit repays itself constantly: read every vertical bar as “given”, and check what is on its right. \(p(y \mid \theta)\) and \(p(\theta \mid y)\) are built from the same joint model but answer opposite questions. Bayesian and frequentist methods differ mainly in what they condition on — the Bayesian conditions on the data actually observed and treats \(\theta\) as random; the frequentist conditions on \(\theta\) and treats the data as random.

Hint