The Place of Model Checking in Applied Bayesian Statistics#

Part 2 · Stage 6 · 🔍 Model Checking & Comparison · Lesson 040 of 144 · intermediate

◀ Previous · Weakly Informative Priors for Variance Parameters · Next · Do the Inferences from the Model Make Sense? ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The third step, at last#

Part I was about steps one and two: build a full probability model, condition on data. Everything downstream — every posterior mean, every credible interval, every decision — is conditional on the model being adequate. Step three asks whether it is. This is not an afterthought appended to a completed analysis; it is what makes the analysis trustworthy, and it is where Part II begins.

Bayes’ rule cannot check itself#

Conditioning is a closed operation: given a model, the posterior follows deterministically. If the likelihood is wrong, Bayes’ rule reports a coherent, confident, wrong posterior, and as Stage 4 showed, more data make it more confident without making it less wrong. Nothing inside the model signals the problem. The check must therefore compare the model’s implications against something outside it — the data, and what you know about the world.

Three questions#

Model checking asks, in ascending order of ambition:

Do the inferences make sense? Are parameter values physically possible, effects of plausible magnitude, signs in the right direction?
Does the model fit the data? Would data generated by the fitted model resemble the data actually observed — in the features you care about? (This is the posterior predictive check, next lesson.)
How sensitive are the conclusions? Would a different reasonable prior, likelihood or subset of the data change what you would do?

A model is a hypothesis, not a truth#

Every model is false; the question is whether it is false in a way that matters for the purpose at hand. So checking is purposive: a model may be perfectly adequate for estimating a mean and hopeless for predicting extremes. Failures are not embarrassments but discoveries — each rejected check points at a missing feature, and the response is to expand the model, refit, and check again. The three steps are a loop.

import arviz as az
# the checking toolkit of the coming lessons, in one place
az.plot_ppc(idata)          # replicated vs observed data
az.loo(idata)               # leave-one-out predictive accuracy
az.plot_loo_pit(idata)      # calibration of predictive distributions

Not hypothesis testing#

A final distinction. Checking is not a significance test that accepts or rejects the model; a \(p\)-value that looks extreme tells you where the model misses, not whether to “reject” it at 5%. The aim is not a verdict but understanding — and, ultimately, a better model.

Hint