Weakly Informative Priors for Variance Parameters#

Part 1 · Stage 5 · 🏛️ Hierarchical Models · Lesson 039 of 144 · beginner

◀ Previous · Hierarchical modeling applied to a meta-analysis · Next · The Place of Model Checking in Applied Bayesian Statistics ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The parameter that decides everything#

In a hierarchical model, \(\tau\) — the group-level standard deviation — governs how much the groups pool. It is also the parameter the data constrain least, especially when the number of groups \(J\) is small (eight schools: \(J = 8\)). So the prior on \(\tau\) is not a formality; it can decide the answer.

Why the old default failed#

For years the reflex was an inverse-gamma**(:math:`epsilon, epsilon`) prior on :math:`tau^2`, with :math:`epsilon = 0.001`, chosen because it is conjugate and “nearly noninformative”. Gelman’s 2006 analysis showed it is neither. As :math:`epsilon to 0` the prior approaches an improper limit whose posterior may not exist; and for any small :math:`epsilon` the prior has **almost no mass near zero while placing weight far out in the tail. When the data genuinely suggest \(\tau \approx 0\) — the eight-schools case — this prior fights them, inflating \(\tau\) and under-pooling. Worse, the answer is sensitive to \(\epsilon\), a number chosen for its irrelevance.

What to use instead#

Put the prior on the standard deviation \(\tau\), not the variance, and choose a density that is positive at zero and has a finite scale:

half-normal — \(\tau \sim \mathrm{N}^{+}(0, s)\), light tails, a sensible default when a rough scale is known;
half-Cauchy — \(\tau \sim \mathrm{C}^{+}(0, s)\), heavy-tailed, allowing large \(\tau\) if the data insist while still regularising;
uniform on \((0, A)\) — acceptable when \(J\) is large and \(A\) is generous.

import pymc as pm
with pm.Model():
    tau = pm.HalfNormal("tau", sigma=5)     # or pm.HalfCauchy("tau", beta=5)
    # NOT: pm.InverseGamma("tau2", alpha=0.001, beta=0.001)
    ...

Scale, and honesty#

The scale \(s\) is a real statement: it should be large relative to the plausible group-level variation and small relative to absurdity. On SAT points (sd ≈ 100) a half-Cauchy(0, 5) says between-school effects of a few points are ordinary and of fifty points are not. Standardise, or scale the prior to the data.

Two closing notes. Allowing \(\tau\) to be near zero is a feature — the model can discover that complete pooling is right — but it creates the funnel geometry that demands the non-centred parameterisation. And with very small \(J\) (say \(J \le 5\)), no prior is truly weak: report the sensitivity, because the prior is doing visible work.

Hint