🔁 Bayesian Decision Theory (BDT)

Bayesian Decision Theory (BDT)#

Choosing the action that minimises expected loss under the posterior distribution.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Core idea#

Bayesian inference hands you a posterior \(p(\theta \mid D)\) over unknown parameters \(\theta\) given data \(D\) — but in practice you don’t just want probabilities, you need to act: classify the email, approve the loan, treat the patient. Bayesian Decision Theory (BDT) turns the posterior into an optimal decision under uncertainty.

The three ingredients#

Actions \(a\) — the choices available (label spam vs not-spam).
States of nature \(\theta\) — the unknown truth (the email really is or isn’t spam).
Loss function \(L(a, \theta)\) — the cost of taking action \(a\) when the truth is \(\theta\) (flagging real mail as spam may cost far more than missing a spam).

Bayes risk and the optimal rule#

The Bayes risk of an action is its posterior expected loss,

\[R(a \mid D) = \mathbb{E}_{\theta \sim p(\theta \mid D)}\big[L(a, \theta)\big],\]

and the Bayes action is the one that minimises it:

\[a^*(D) = \arg\min_a R(a \mid D).\]

By construction this is the choice that does best on average under everything the posterior knows.

The classification special case#

Let \(\theta \in \{C_1, \dots, C_k\}\) with posterior class probabilities \(p(C_i \mid x)\). Under 0–1 loss (0 if correct, 1 if wrong), the Bayes-optimal classifier reduces to

\[a^*(x) = \arg\max_i \; p(C_i \mid x),\]

which is exactly the MAP (maximum a posteriori) classifier — pick the most probable class.

Asymmetric loss shifts the boundary#

When errors cost differently, BDT moves the decision threshold rather than the 0.5 default. In a medical test where a missed disease (false negative) is worse than a false alarm, the optimal rule classifies as positive at a lower posterior probability — trading more false alarms for fewer missed cases.

Where it shows up#

BDT is the formal backbone of decision-making under uncertainty across ML (classification, regression, model selection), medicine (treat vs not), finance (portfolio choice under risk) and engineering / reinforcement learning — including bandit strategies like Thompson Sampling, which is BDT applied sequentially with the posterior updated as rewards arrive.

Theme: Bayesian Inference · All terminology

Hint

Mind map — connected ideas

Thompson Sampling (TS) in Bandits (Multi-Armed Bandit Problem (MAB)) · Loss Functions · Posterior · Bayes’ Theorem · Posterior Probability · Bandit Algorithms

Hint

More in Bayesian Inference

Bayes’ Theorem · Bayesian Correction · Bayesian Inference. · Bayesian Neural Networks (BNNs) · Binomial Likelihood · Gaussian Processes (GPs) · Marginal Likelihood (also called The Model Evidence or Integrated Likelihood) · MCMC (Markov Chain Monte Carlo) · Parameter(s) of Interest · Posterior · Posterior belief · Posterior Probability · Posterior probability of uplift · Prior Belief (or Prior Probability)