🔁 Marginal Likelihood (also called The Model Evidence or Integrated Likelihood)

Marginal Likelihood (also called The Model Evidence or Integrated Likelihood)#

The probability of the data averaged over the prior — the normaliser and a model-comparison score.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

The marginal likelihood — also called the model evidence or integrated likelihood — is the probability of the observed data under a model, averaged over all possible parameter values. It is the denominator in Bayes’ theorem:

\[P(\theta \mid D) = \frac{P(D \mid \theta)\, P(\theta)}{P(D)}, \qquad P(D) = \int P(D \mid \theta)\, P(\theta)\, d\theta.\]

In words, \(P(D)\) is the prior-weighted average of the likelihood — the overall probability of the data, considering every parameter value the prior allows.

Two jobs#

1. Normalisation. Dividing by \(P(D)\) is what makes the posterior integrate to 1 — without it, prior × likelihood is only proportional to the posterior.

2. Model comparison. Because it scores how well an entire model (not one parameter setting) predicts the data, the marginal likelihood is the basis of the Bayes factor,

\[\text{BF} = \frac{P(D \mid M_1)}{P(D \mid M_2)},\]

which weighs two competing models — a higher marginal likelihood means the model explains the data better, with a built-in Occam penalty for needless complexity.

Example — coin toss#

With a uniform prior \(\text{Beta}(1,1)\) and 7 heads in 10 tosses,

\[P(D) = \int_0^1 \binom{10}{7} p^7 (1-p)^3 \, dp,\]

the overall probability of “7 heads out of 10” averaged across all plausible \(p\).

Why it’s hard#

That integral is usually intractable in real models, which is the whole reason approximate methods exist: the Laplace approximation, variational inference (whose ELBO is a lower bound on \(\log P(D)\)), and MCMC-based estimators all exist to approximate the evidence for posterior computation and model selection.

Theme: Bayesian Inference · All terminology

Hint

Mind map — connected ideas

Posterior · Prior Belief (or Prior Probability) · Bayes’ Theorem · Variational Inference (VI) · Binomial Likelihood

Hint

More in Bayesian Inference

Bayes’ Theorem · Bayesian Correction · Bayesian Decision Theory (BDT) · Bayesian Inference. · Bayesian Neural Networks (BNNs) · Binomial Likelihood · Gaussian Processes (GPs) · MCMC (Markov Chain Monte Carlo) · Parameter(s) of Interest · Posterior · Posterior belief · Posterior Probability · Posterior probability of uplift · Prior Belief (or Prior Probability)