Setting up and interpreting mixture models#

Part 5 · Stage 16 · ♾️ Mixtures & Nonparametric Bayes · Lesson 134 of 144 · advanced

◀ Previous · Density estimation and regression · Next · Example: reaction times and schizophrenia ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The latent-class formulation#

A mixture becomes easy to fit once written with an explicit membership indicator. Introduce, for each observation, an unobserved label \(z_i\) saying which component generated it:

\[z_i \sim \mathrm{Categorical}(\pi), \qquad y_i \mid z_i = k \sim \mathrm{N}(\mu_k, \sigma_k^2).\]

Marginalising \(z_i\) recovers the weighted-sum density of the previous lesson, but keeping \(z_i\) as a latent variable turns an awkward sum inside the likelihood into a two-level model with simple conditionals — the same data-augmentation trick used for the \(t\) distribution and for probit regression.

Fitting: EM and Gibbs#

The augmented form yields two classic algorithms, both alternating between the labels and the parameters. EM (from the modal-approximation stage) computes, in its E-step, the posterior responsibilities \(\Pr(z_i = k \mid y_i)\) — soft assignments — then in the M-step updates \((\pi, \mu, \sigma)\) given them. Gibbs does the Bayesian version: sample each \(z_i\) from its conditional, then sample the parameters given the assignments, and repeat.

import numpy as np
# EM for a Gaussian mixture: responsibilities, then weighted updates
for _ in range(n_iter):
    # E-step: responsibility of component k for point i
    r = pi * norm_pdf(y[:, None], mu, sigma)             # (n, K)
    r /= r.sum(axis=1, keepdims=True)
    # M-step: update weights, means, sds from soft assignments
    Nk = r.sum(axis=0)
    pi = Nk / len(y)
    mu = (r * y[:, None]).sum(0) / Nk
    sigma = np.sqrt((r * (y[:, None] - mu)**2).sum(0) / Nk)

Interpreting the fit#

The responsibilities are the useful output when the goal is clustering: each \(\Pr(z_i = k)\) is a soft membership, honestly expressing that a point between two components belongs partly to each — better than a hard assignment that hides the ambiguity. When the goal is density estimation, the labels are a computational device and only the resulting smooth density matters.

Cautions#

Two recur through the stage. Mixture likelihoods are multimodal — genuinely, beyond label switching — so optimisation and sampling can stick in local solutions; multiple starts and careful diagnostics are needed. And a component’s variance can collapse toward zero as it latches onto a single point, sending the likelihood to infinity — the same boundary pathology met in hierarchical models, cured the same way, with a prior that keeps variances off zero. The next lesson puts the setup to work; the one after confronts label switching head-on.

Hint