Model comparison using Bayes factors#

Part 2 · Stage 6 · 🔍 Model Checking & Comparison · Lesson 047 of 144 · intermediate

◀ Previous · Model comparison based on predictive performance · Next · Continuous model expansion ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Betting on whole models#

If the candidate models are treated as hypotheses with prior probabilities, Bayes’ rule applies to them as well. The Bayes factor is the ratio of the two marginal likelihoods — the evidence term that inference could ignore, now doing all the work:

\[\mathrm{BF}_{12} = \frac{p(y \mid M_1)}{p(y \mid M_2)}, \qquad p(y \mid M_k) = \int p(y \mid \theta_k, M_k) \; p(\theta_k \mid M_k) \; d\theta_k .\]

It multiplies the prior odds into posterior odds. Because the marginal likelihood averages the likelihood over the prior, it automatically penalises models that spread their prior mass over regions the data reject — an intrinsic Occam’s razor, with no explicit parameter count.

Three difficulties#

That elegance carries a heavy bill.

Improper priors make it undefined. An improper prior has an arbitrary normalising constant, which does not cancel; the Bayes factor takes an arbitrary value.
Vague proper priors favour the simpler model — the Jeffreys–Lindley (Bartlett) paradox. Widening a prior dilutes the alternative’s marginal likelihood, so a diffuse prior on the alternative drives the Bayes factor toward the null no matter what the data say, and the effect does not vanish as \(n\) grows.
It is hard to compute. The marginal likelihood is exactly the normalising constant that MCMC is built to avoid, and naive estimators of it are notoriously bad.

The sensitivity is the real objection#

Notice what makes this different from ordinary prior sensitivity. Changing \(\mathrm{Beta}(1,1)\) to \(\mathrm{Beta}(30,30)\) may barely move the posterior for \(\theta\) — but it can change the Bayes factor substantially, because the evidence integrates the likelihood against the prior itself. A quantity that is insensitive where inference is sensitive, and sensitive where inference is not, is a poor guide.

# Bayes factors demand priors you would defend as *predictions*, not as regularisers.
# In practice, prefer:
import arviz as az
az.compare({"m1": idata1, "m2": idata2})   # predictive comparison, no marginal likelihood

When to use it#

Bayes factors are appropriate when the model space is genuinely discrete and exhaustive — one of these hypotheses is true — and the priors are honest, informative statements you would stake a prediction on. That describes some scientific hypothesis tests and few applied models. Gelman’s recommended route is to bypass the choice: check models predictively, compare them by elpd, and, where they disagree, expand rather than select — the subject of the next lesson.

Hint