Label switching and posterior computation#

Part 5 · Stage 16 · ♾️ Mixtures & Nonparametric Bayes · Lesson 136 of 144 · advanced

◀ Previous · Example: reaction times and schizophrenia · Next · Unspecified number of mixture components ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The symmetry problem#

Mixture models carry a computational pathology absent from everything before them. If the components share a prior and come from the same family, then permuting the component labels leaves the model unchanged — calling component 1 “component 2” and vice versa gives an identical likelihood and prior. The posterior is therefore invariant under all \(K!\) relabellings, and this is label switching.

Why it breaks inference#

The posterior has \(K!\) identical modes, one per permutation. A sampler that mixes well visits all of them, so a chain’s draws for “the mean of component 1” jump between what are really different components — averaging them gives the overall mean of all components, a meaningless number. Any inference that refers to a component by its label is corrupted: the marginal posterior of \(\mu_1\) is identical to that of \(\mu_2\), both equal to the mixture of all components’ means.

import numpy as np
# symptom: per-component means are identical across the chain because labels swap
mu = idata.posterior["mu"].values.reshape(-1, K)
mu.mean(axis=0)          # all K entries nearly equal  <-  label switching

Solutions#

Three standard remedies, in rough order of robustness:

Ordering constraint. Impose an identifying restriction — \(\mu_1 < \mu_2 < \cdots\) or ordered weights — that picks one labelling. Simple and often enough, as the reaction-time delay (\(\tau > 0\)) showed; but a poor choice of ordering variable can distort a genuinely multimodal posterior.
Post-hoc relabelling. Let the sampler run unconstrained, then permute each draw to a common labelling by solving an assignment problem — matching components across draws by their parameters. More flexible than a hard constraint and the standard modern approach.
Label-invariant summaries. Report only quantities that do not depend on labels — the density \(p(y)\), the number of components, the clustering of observations (which points group together) — all unchanged by permutation.

The deeper lesson#

Which fix to use follows from the goal. For density estimation, label switching is a non-problem: the estimated density is label-invariant, so an unconstrained chain gives exactly the right answer with nothing to fix. It bites only for component-specific interpretation — “what is the mean of the second group?” — which presupposes the components are real and distinguishable. So the presence of a label-switching problem is a signal to ask whether component-level inference is even meaningful for the question, or whether a label-invariant summary answers it directly.

Hint