Frequency Evaluations of Bayesian Inferences#

Part 1 · Stage 4 · 📏 Asymptotics & Frequentist Ties · Lesson 031 of 144 · beginner

◀ Previous · Counterexamples to large-sample (asymptotic) Bayesian theorems · Next · Bayesian interpretations of other statistical methods ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

A different question#

Bayesian inference conditions on the data you have. Frequency evaluation asks a different, entirely legitimate question: if this procedure were applied repeatedly, across many datasets, how would it behave? The two views are not rivals — the frequency properties of a Bayesian method are a way of checking it, and Gelman treats them as diagnostics rather than definitions.

Coverage#

The headline criterion is calibration of intervals: do 95% credible intervals contain the true parameter 95% of the time? Bernstein–von Mises promises this asymptotically. In small samples the answer depends on the prior — and often, informatively, in the Bayesian method’s favour: a weakly informative prior that shrinks noisy estimates can achieve better coverage and much smaller average error than an unbiased estimator, particularly in the many-parameter, small-\(n\) settings (the cancer-rate map) where classical methods flounder.

Simulate to check#

The evaluation is mechanical: draw parameters from the prior, simulate data from the model, fit, and count. This simulation-based calibration checks the prior, the likelihood and the sampler together — if a 50% interval covers 70% of the time, something is wrong somewhere.

import numpy as np
from scipy import stats
rng = np.random.default_rng(0)
n, a, b, cover = 20, 2, 8, 0
for _ in range(2000):
    theta = stats.beta(a, b).rvs(random_state=rng)      # draw from the prior
    y = stats.binom(n, theta).rvs(random_state=rng)     # simulate data
    lo, hi = stats.beta(a + y, b + n - y).interval(0.95)
    cover += lo <= theta <= hi
cover / 2000        # ≈ 0.95 when prior, model and computation all agree

Bias, variance, and honesty#

Bayesian estimators are typically biased — that is what shrinkage means — and typically achieve lower mean squared error for it, trading a little bias for a lot of variance. Frequency evaluation makes the bargain explicit rather than hiding it. Two honest limits: the calibration above averages over the prior, so it certifies the procedure only if that prior is the one you believe; and no amount of frequency checking rescues a misspecified likelihood. Coverage is a necessary condition for trust, not a sufficient one.

Hint