Boundary-avoiding priors for modal summaries#

Part 3 · Stage 10 · 🎛️ Modal & Variational Approximation · Lesson 082 of 144 · intermediate

◀ Previous · Finding posterior modes · Next · Normal and related mixture approximations ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Zero variance, implausibly#

Fit a hierarchical model by maximum likelihood with only a few groups, and a familiar pathology appears: the estimated group-level variance is exactly zero. Not small — zero. The likelihood’s maximum lies on the boundary of the parameter space, and the software reports complete pooling as a certainty. Anyone who believes the groups differ at all knows the estimate is wrong.

Why it happens#

With \(J\) small, the between-group variation observed in the data can easily be no larger than what sampling noise alone would produce. The likelihood for \(\tau\) is then monotonically decreasing, so its maximum is at \(\tau = 0\). Nothing is broken; the mode is simply a bad summary of a posterior that has plenty of mass at positive \(\tau\). Full Bayes avoids the problem by averaging rather than maximising — but modal estimation is popular for speed, so it needs a fix of its own.

The fix: penalise the boundary#

Chung, Rabe-Hesketh, Dorie, Gelman and Liu proposed a boundary-avoiding prior: put a \(\mathrm{Gamma}(\alpha, \lambda)\) prior on the group-level standard deviation with shape \(\alpha > 1\), so that the density vanishes at zero and the posterior mode cannot land there.

\[p(\tau) \propto \tau^{\alpha - 1} e^{-\lambda \tau}, \qquad \alpha = 2, \; \lambda \to 0 .\]

With \(\alpha = 2\) the prior is proportional to \(\tau\) near the origin — enough to push the mode off the boundary, not enough to fight real data. The default \(\mathrm{Gamma}(2, \lambda \to 0)\) has three properties worth stating precisely: the penalised estimate is approximately one standard error from zero when the ML estimate is zero; it closely approximates the posterior median under a noninformative prior; and as the number of groups grows it coincides with maximum likelihood. The prior does work exactly where the data are silent, and withdraws where they speak.

import pymc as pm
with pm.Model():
    # boundary-avoiding: density -> 0 as tau -> 0, so the MODE stays positive
    tau = pm.Gamma("tau", alpha=2, beta=1e-4)
    # (for full-Bayes sampling, HalfNormal/HalfCauchy remain the default;
    #  boundary-avoidance is specifically about modal summaries)
    ...

Covariance matrices too#

The same authors extended the idea to group-level covariance matrices, where maximum likelihood routinely returns a singular estimate: use a Wishart prior — deliberately not the inverse-Wishart of Stage 3 — with degrees of freedom equal to the number of varying coefficients plus two, which keeps the modal estimate positive definite.

The moral generalises. A prior chosen for modal estimation is not the prior you would choose for sampling: it must shape the density’s peak, not merely its mass. Naming which summary you intend is part of specifying the model.

Hint