Density estimation and regression#

Part 5 · Stage 16 · ♾️ Mixtures & Nonparametric Bayes · Lesson 133 of 144 · advanced

◀ Previous · Functional data analysis · Next · Setting up and interpreting mixture models ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Flexibility from components, not curves#

Part V has pursued flexibility through smooth functions — splines, Gaussian processes. The final stage takes a different route: build flexible distributions and regressions by combining simple components. This lesson frames the two problems the stage solves, and the shift in thinking that unites them.

Two nonparametric problems#

Density estimation asks for the whole distribution of a variable, \(p(y)\), when no parametric family fits — a distribution with several bumps, heavy skew, or unknown shape. Density regression goes further: it lets the entire conditional distribution \(p(y \mid x)\) change with predictors, not merely its mean. A standard regression models \(\mathrm{E}[y \mid x]\) and assumes the rest of the shape is fixed; density regression frees the variance, the skew, even the number of modes to vary with \(x\).

The mixture idea#

Both are solved by mixtures. Any sufficiently complex distribution can be approximated by a weighted sum of simple components — usually normals:

\[p(y) = \sum_{k=1}^{K} \pi_k \, \mathrm{N}(y \mid \mu_k, \sigma_k^2), \qquad \sum_k \pi_k = 1 .\]

With enough components a mixture of normals can approximate any continuous density to arbitrary accuracy — the theoretical guarantee that makes mixtures a general-purpose nonparametric tool. The components need not correspond to real subpopulations; they are basis elements for building a shape.

import numpy as np, pymc as pm
with pm.Model():
    w = pm.Dirichlet("w", a=np.ones(K))                  # mixture weights, sum to 1
    mu = pm.Normal("mu", 0, 5, shape=K)
    sigma = pm.HalfNormal("sigma", 1, shape=K)
    pm.NormalMixture("y", w=w, mu=mu, sigma=sigma, observed=y)   # flexible density

Two readings, and the plan#

Mixtures carry a productive ambiguity. Model-based clustering reads each component as a real subpopulation — the mixture discovers groups, and the membership of each point is the inference of interest. Density estimation reads the components as mere building blocks for an arbitrary shape, with no claim that they are real. The same model, two purposes; which one you intend governs how you interpret the fit. The stage builds from finite mixtures (this lesson and the next few) to the case of an unknown number of components, and finally to infinite mixtures — the Dirichlet process — where the component count is itself learned. Flexibility, assembled from simple parts.

Hint