Multinomial Model for Categorical Data#

Part 1 · Stage 3 · 🧮 Multiparameter Models · Lesson 023 of 144 · beginner

◀ Previous · Normal Data with a Conjugate Prior Distribution · Next · Multivariate Normal Model with Known Variance ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Beyond two categories#

Generalise the binomial from two outcomes to \(k\). Each observation falls in exactly one of \(k\) categories with probabilities \(\theta = (\theta_1, \dots, \theta_k)\), \(\sum_j \theta_j = 1\), and the counts \(y = (y_1, \dots, y_k)\) follow a multinomial likelihood:

\[p(y \mid \theta) \;\propto\; \prod_{j=1}^{k} \theta_j^{\,y_j} .\]

Poll responses, survey categories, and the components of the mixture models in Part V all live here.

The Dirichlet prior#

The conjugate prior on the simplex is the Dirichlet, the Beta’s multivariate sibling: \(\theta \sim \mathrm{Dirichlet}(\alpha_1, \dots, \alpha_k)\), with density proportional to \(\prod_j \theta_j^{\,\alpha_j - 1}\). Same functional form as the likelihood — so the update is again pure counting:

\[\theta \mid y \;\sim\; \mathrm{Dirichlet}(\alpha_1 + y_1,\; \dots,\; \alpha_k + y_k).\]

Each \(\alpha_j\) is a prior count in category \(j\), and \(\sum_j \alpha_j\) is the prior sample size. \(\mathrm{Dirichlet}(1, \dots, 1)\) is uniform on the simplex; Jeffreys’ choice is all \(\alpha_j = \tfrac12\).

Contrasts come free#

The real payoff is that questions about differences are answered directly from draws — no delta method, no covariance algebra:

import numpy as np
y = np.array([420, 380, 200])            # candidate A, B, undecided
alpha = np.ones(3)                       # uniform prior
draws = np.random.dirichlet(alpha + y, size=20_000)
lead = draws[:, 0] - draws[:, 1]         # posterior of the A - B margin
lead.mean(), (lead > 0).mean()           # P(A leads B | data) ≈ 0.91

Two structural facts#

Marginally, each \(\theta_j\) is \(\mathrm{Beta}(\alpha_j,\ \alpha_0 - \alpha_j)\) — collapse the other categories and the binomial reappears. And the components are negatively correlated by construction: they must sum to one, so probability given to one category is taken from another. That constraint is exactly what makes the simplex the natural home for mixture weights in Stage 16.

Hint