🔁 Bayesian Neural Networks (BNNs)

Bayesian Neural Networks (BNNs)#

Neural networks with distributions over their weights, producing predictive uncertainty.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

From point estimates to distributions#

A standard neural network fixes its weights \(W\) after training — gradient descent finds a single best point estimate. That captures no uncertainty: on limited or noisy data the network can be confidently wrong. A Bayesian neural network (BNN) fixes this by treating the weights as random variables with distributions.

The Bayesian view of weights#

Instead of one weight vector, infer a posterior over weights:

\[p(W \mid D) \propto p(D \mid W)\, p(W),\]

with prior \(p(W)\) (often a zero-mean Gaussian), likelihood \(p(D \mid W)\) (how well the weights explain the data) and posterior \(p(W \mid D)\). Training therefore yields a whole distribution of plausible networks, not one.

Prediction integrates over weights#

Predictions average over every weight setting, weighted by posterior probability:

\[p(y \mid x, D) = \int p(y \mid x, W)\, p(W \mid D)\, dW.\]

This separates two kinds of uncertainty: aleatoric (noise in the labels) and epistemic (not enough data to pin down the weights) — and the output is a distribution over predictions, not a point.

Why it’s hard, and how it’s done#

That integral is intractable, so BNNs rely on approximation:

Variational inference — approximate the posterior with a simpler \(q(W)\).
MCMC — sample weights from the posterior (accurate but slow).
MC Dropout (Gal & Ghahramani, 2016) — keeping dropout on at test time approximates sampling from a weight distribution, a cheap practical trick.

Why it matters#

A BNN knows when it doesn’t know: it resists overconfident errors and generalises better, which is decisive in medicine, self-driving and finance where the cost of an uncertain prediction matters. The price is heavier training and inference, sensitivity to the approximation chosen, and less mainstream tooling than standard nets.

Theme: Bayesian Inference · All terminology

Hint

Mind map — connected ideas

Variational Inference (VI) · Gaussian Processes (GPs) · MCMC (Markov Chain Monte Carlo) · Posterior · Bayesian Inference.

Hint

More in Bayesian Inference

Bayes’ Theorem · Bayesian Correction · Bayesian Decision Theory (BDT) · Bayesian Inference. · Binomial Likelihood · Gaussian Processes (GPs) · Marginal Likelihood (also called The Model Evidence or Integrated Likelihood) · MCMC (Markov Chain Monte Carlo) · Parameter(s) of Interest · Posterior · Posterior belief · Posterior Probability · Posterior probability of uplift · Prior Belief (or Prior Probability)