🔁 Variational Inference (VI)

Variational Inference (VI)#

Approximating an intractable posterior by optimising a simpler distribution to be close to it.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The problem it solves#

Bayesian inference wants the posterior

\[p(\theta \mid x) = \frac{p(x \mid \theta)\, p(\theta)}{p(x)},\]

but the evidence in the denominator, \(p(x) = \int p(x \mid \theta)\, p(\theta)\, d\theta\), is an integral over all parameters that is intractable in most real models. MCMC tackles this by sampling; variational inference (VI) tackles it by optimisation.

The core idea#

Replace the hard posterior with the closest member of a simpler family \(q(\theta)\) (say, Gaussians):

\[q^*(\theta) = \arg\min_q \; \operatorname{KL}\!\big(q(\theta) \,\Vert\, p(\theta \mid x)\big),\]

measuring closeness by Kullback–Leibler divergence. Inference becomes a search for the best-fitting approximation.

The ELBO#

We can’t minimise that KL directly (it contains the unknown posterior), so VI instead maximises the Evidence Lower Bound:

\[\mathcal{L}(q) = \mathbb{E}_{q(\theta)}\big[\log p(x, \theta) - \log q(\theta)\big].\]

Maximising the ELBO is equivalent to minimising the KL, and the ELBO is a genuine lower bound on \(\log p(x)\) — so gradient-based optimisation of \(\mathcal{L}\) drives \(q\) toward the posterior.

How it’s done in practice#

Pick a variational family \(q(\theta; \phi)\) (e.g. a Gaussian’s mean and variance), then optimise \(\phi\) to maximise the ELBO and use \(q\) as the posterior. Common machinery:

Mean-field — assume independence, \(q(\theta) = \prod_i q_i(\theta_i)\).
CAVI — coordinate ascent, updating each factor in turn.
Stochastic VI — mini-batch stochastic optimisation for large data.
Reparameterisation trick — write \(\theta = g(\phi, \epsilon)\) with noise \(\epsilon\) so gradients flow through samples (the engine of variational autoencoders).

Where it shows up#

Topic models (latent Dirichlet allocation), variational autoencoders, Bayesian neural networks and probabilistic graphical models. The trade-off vs MCMC: VI is faster and scalable but gives a biased approximation (only as good as the family), whereas MCMC is asymptotically exact but slower.

Theme: Bayesian Inference · All terminology

Hint

Mind map — connected ideas

MCMC (Markov Chain Monte Carlo) · Bayesian Neural Networks (BNNs) · Kullback–Leibler (KL) Divergence · Posterior · Bayesian Inference.

Hint

More in Bayesian Inference

Bayes’ Theorem · Bayesian Correction · Bayesian Decision Theory (BDT) · Bayesian Inference. · Bayesian Neural Networks (BNNs) · Binomial Likelihood · Gaussian Processes (GPs) · Marginal Likelihood (also called The Model Evidence or Integrated Likelihood) · MCMC (Markov Chain Monte Carlo) · Parameter(s) of Interest · Posterior · Posterior belief · Posterior Probability · Posterior probability of uplift