Interpreting a normal prior distribution as extra data#

Part 4 · Stage 12 · 🏗️ Hierarchical Regression · Lesson 101 of 144 · advanced

◀ Previous · Example: forecasting U.S. presidential elections · Next · Varying intercepts and slopes ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

A prior is imaginary data#

The previous lesson added quantitative prior knowledge through a prior on \(\beta\). There is an exact algebraic identity underneath it, and it is one of the most clarifying facts in Bayesian regression: a normal prior on the coefficients is equivalent to a set of extra data points appended to the regression.

The augmented-data identity#

Take the linear model with a normal prior \(\beta \sim \mathrm{N}(\beta_0, \Sigma_0)\). Its posterior mode is identical to the least-squares fit of an augmented dataset — the real observations, plus one pseudo-observation per prior constraint:

\[\begin{split}\bar{X} = \begin{bmatrix} X \\ \Sigma_0^{-1/2} \end{bmatrix}, \qquad \bar{y} = \begin{bmatrix} y \\ \Sigma_0^{-1/2}\beta_0 \end{bmatrix}, \qquad \bar{X}^{\top}\bar{X} = \underbrace{X^{\top}X}_{\text{data}} + \underbrace{\Sigma_0^{-1}}_{\text{prior}} .\end{split}\]

The precision decomposes into a data term and a prior term that simply add. Each augmented row is one imaginary observation stating “at this design point the response was \(\beta_0\)”, carrying a precision set by the prior. The special case \(\beta_0 = 0\), \(\Sigma_0 = (\sigma^2/\lambda) I\) recovers ridge regression exactly — appending \(\sqrt{\lambda}\, e_j\) rows with response zero.

import numpy as np
# prior beta ~ N(beta0, diag(tau^2)) as pseudo-observations, then plain least squares
P = np.diag(1.0 / tau)                            # Sigma0^{-1/2}
X_aug = np.vstack([X, P])
y_aug = np.concatenate([y, P @ beta0])
beta_post_mode = np.linalg.lstsq(X_aug, y_aug, rcond=None)[0]   # == posterior mode

Why the picture helps#

It makes the strength of a prior concrete. A tight prior is many pseudo-observations; a vague one is few; their precisions add to the data’s exactly as a second dataset would. It demystifies regularisation — ridge, and its relatives, are priors and nothing more. And it clarifies the balance of evidence: where the real data are informative about a coefficient, they swamp the handful of pseudo-rows; where they are silent (collinearity, few observations), the prior carries the estimate, which is precisely when you want it to.

The counting interpretation#

The identity even assigns the prior an effective sample size. A prior with precision \(\Sigma_0^{-1}\) contributes as much information as that many real observations at the corresponding design points, so “how strong is this prior?” has a literal answer in units of data. That is the honest way to feel the weight of a prior — and the honest warning against a prior so tight it silently adds hundreds of observations you never collected.

Hint