Do the Inferences from the Model Make Sense?#

Part 2 · Stage 6 · 🔍 Model Checking & Comparison · Lesson 041 of 144 · intermediate

◀ Previous · The Place of Model Checking in Applied Bayesian Statistics · Next · Posterior predictive checking ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The cheapest check#

Before any simulation, before any \(p\)-value, look at the answer and ask whether it is possible. This costs nothing and catches a startling share of real errors — coding mistakes, unit confusions, mis-specified priors, sign flips. External knowledge that never entered the model is exactly what makes the check informative.

What to look at#

Read the posterior against what you already know:

Magnitude. Is a coaching effect of 28 SAT points plausible, given that the test is designed to resist short-term preparation? Is an odds ratio of 400 credible, or a symptom of separation?
Sign and direction. Does the coefficient point the way theory demands? A negative dose effect in a toxicity assay is a red flag, not a discovery.
Physical constraints. Probabilities in \([0,1]\), variances positive, rates non-negative, populations integral.
Derived quantities. Push draws through the functions you care about — an LD50, a predicted count, a break-even price — and see whether the implied statements are absurd.
Extremes. Look at the widest intervals and the most-shrunken groups. Do they behave as the structure of the data predicts?

External validation#

The strongest version compares model output with information deliberately excluded from the fit: held-out data, a later replication, an independent measurement, a published estimate. Agreement is weak evidence for the model; sharp disagreement is strong evidence against it, and localises the fault.

import numpy as np, arviz as az
post = idata.posterior
# 1. do parameters respect their constraints?
assert float(post["tau"].min()) >= 0
# 2. do derived quantities land in plausible ranges?
ld50 = -post["alpha"] / post["beta"]
np.percentile(ld50, [2.5, 50, 97.5])       # compare with known chemistry
# 3. does the model's implied prediction match a value you did not fit to?
az.summary(idata, var_names=["mu"])        # against the published meta-analytic estimate

Being wrong usefully#

An implausible inference has three possible causes, in decreasing order of frequency: a bug, a bad prior, or a wrong likelihood. Rule them out in that order — check the data pipeline and units first, then simulate from the prior, then question the model. And keep the caveat honest: a result that merely surprises you may be the finding. The check is against what is impossible or incoherent, not against what is merely unexpected.

Hint