Bayesian interpretations of other statistical methods#

Part 1 · Stage 4 · 📏 Asymptotics & Frequentist Ties · Lesson 032 of 144 · beginner

◀ Previous · Frequency Evaluations of Bayesian Inferences · Next · Constructing a Parameterized Prior Distribution ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Priors in disguise#

Many familiar non-Bayesian procedures turn out to be Bayesian estimates under some prior. Recognising this is not point-scoring: it clarifies what a method assumes, and it converts tuning parameters chosen by cross-validation into statements about prior beliefs — which can then be examined, criticised and improved.

Penalised likelihood is MAP estimation#

Maximising a penalised log-likelihood is exactly finding a posterior mode:

\[\underbrace{\arg\max_{\theta} \; \bigl[\log p(y \mid \theta) + \log p(\theta)\bigr]}_{\text{MAP}} \;\;=\;\; \underbrace{\arg\min_{\theta} \; \bigl[-\log p(y \mid \theta) + \lambda\, \mathrm{pen}(\theta)\bigr]}_{\text{penalised likelihood}} ,\]

with the penalty as the negative log prior. Two headline cases:

Ridge regression \(= ` MAP under a **Gaussian** prior :math:\)beta_j sim mathrm{N}(0, tau^2)`, with \(\lambda = \sigma^2 / \tau^2\). The \(\ell_2\) penalty is \(-\log p(\beta)\) up to a constant.
Lasso :math:`= ` MAP under a Laplace (double-exponential) prior, whose peak at zero produces exact zeros at the mode.

Maximum likelihood itself is MAP under a flat prior — which is why, per Bernstein–von Mises, the two agree asymptotically.

Where the analogy stops#

The correspondence is between point estimates, not distributions, and that is the catch. The lasso’s sparse solution is a property of the mode; the full posterior under a Laplace prior puts zero probability on any coefficient being exactly zero. Reporting the mode alone hides this. The Bayesian version supplies uncertainty — and reveals that the sparsity was an artifact of the summary. (Modern Bayesian sparsity uses horseshoe or spike-and-slab priors instead.)

from sklearn.linear_model import Ridge
import numpy as np
# Ridge with alpha = sigma^2 / tau^2 reproduces the Gaussian-prior posterior MEAN
# (for linear-Gaussian models the mode and mean coincide)
Ridge(alpha=1.0).fit(X, y).coef_

Others in the family#

The list extends: shrinkage/empirical-Bayes estimators (James–Stein) are hierarchical posteriors with hyperparameters fitted from data; smoothing splines are Gaussian-process posteriors with a roughness prior (Stage 15); regularised logistic regression is the weakly-informative prior that cures separation (Part IV). The Bayesian reading gives each a language for why it works: not “the penalty stabilises the fit”, but “here is what the analysis assumes about the world” — a claim that can be stated, checked, and defended.

Hint