Using Gibbs and Metropolis as building blocks#

Part 3 · Stage 9 · ⛓️ MCMC: Gibbs, Metropolis & HMC · Lesson 071 of 144 · intermediate

◀ Previous · Metropolis and Metropolis-Hastings algorithms · Next · Inference and assessing convergence ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Mix and match#

Gibbs and Metropolis are not rivals; they are components. A realistic model has some parameters with tidy conjugate conditionals and others without. The natural sampler updates each block by whatever method suits it, and the composition still targets the correct posterior.

Metropolis-within-Gibbs#

Sweep through the parameter blocks as in Gibbs. For a block whose full conditional is a standard distribution, draw from it (acceptance 1). For a block whose conditional is only known up to a constant, take a Metropolis step targeting that conditional. Each update leaves the posterior invariant, so the composed chain does too.

import numpy as np
from scipy import stats
for t in range(n_iter):
    # block 1: conjugate -> exact Gibbs draw
    sigma2 = stats.invgamma(a + n / 2, scale=b + 0.5 * ((y - X @ beta) ** 2).sum()).rvs()
    # block 2: no closed form -> Metropolis step on the conditional
    prop = beta + step * np.random.standard_normal(len(beta))
    if np.log(np.random.rand()) < log_cond(prop, sigma2) - log_cond(beta, sigma2):
        beta = prop

Blocking and reparameterising#

The two levers that fix slow mixing, both aimed at posterior correlation:

Blocking — update strongly dependent parameters jointly rather than one at a time. Gibbs moves along axes; a correlated pair traversed jointly moves along the ridge. Regression coefficients under a conjugate prior can be drawn as a whole vector.
Reparameterising — change coordinates so the posterior is less correlated. Centre predictors; replace \(\theta_j \sim \mathrm{N}(\mu, \tau^2)\) with \(\theta_j = \mu + \tau \eta_j\), \(\eta_j \sim \mathrm{N}(0,1)\). The non-centred parameterisation from eight schools is exactly this move, and it works for Gibbs and HMC alike.

Auxiliary variables#

A third trick: add parameters to make the conditionals tractable. Data augmentation introduces latent variables (a \(t\) distribution written as a scale-mixture of normals; a probit model given latent normals) whose presence turns an awkward joint into a chain of conjugate conditionals. You sample in a larger space and discard the extra columns, which — per Part I — is marginalisation.

The upshot#

These compositions dominated Bayesian computation for two decades, and they still matter: they explain what BUGS and JAGS do, they remain the right tool for discrete parameters (which gradient methods cannot touch), and their diagnostics are the subject of the next lesson. Their limitation is universal — all of them explore by random walk, and none escape its quadratic cost.

Hint