Beyond density estimation#

Part 5 · Stage 16 · ♾️ Mixtures & Nonparametric Bayes · Lesson 142 of 144 · advanced

◀ Previous · Dirichlet process mixtures · Next · Hierarchical dependence ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Nonparametrics for other functionals#

Dirichlet process mixtures were introduced for density estimation, but their reach is far wider. Once you have a flexible posterior over a whole distribution, any functional of that distribution inherits a posterior — so nonparametric Bayes answers questions that are not about the density’s shape at all.

Functionals come free#

A DPM yields posterior draws of the entire distribution \(G\) (or the implied density). Push each draw through any functional and you get its posterior automatically — no new model required:

\[T(G) : \quad \text{mean, variance, quantiles, } \Pr(Y > c), \; \text{entropy, mode count, } \dots\]

Each posterior draw of \(G\) gives one draw of \(T(G)\), so a median, a tail probability, or the number of modes arrives with full uncertainty — including uncertainty about the distributional shape, which a parametric model would have suppressed by assuming it away. Estimating a 99th percentile from a skewed, multimodal distribution is exactly where this pays off.

import numpy as np
# each posterior draw is a full mixture -> evaluate any functional per draw
def functional_posterior(weights_draws, mu_draws, sigma_draws, T):
    return np.array([T(w, m, s)                          # one value per posterior draw
                     for w, m, s in zip(weights_draws, mu_draws, sigma_draws)])
# e.g. T = tail probability P(Y > c), or a quantile, or the number of modes

Model-based clustering as inference#

The DPM’s partition is itself a rich object. Because the model puts a posterior over how the data divide into groups, clustering stops being a point estimate from an algorithm and becomes inference: you get the posterior probability that two points share a cluster, the distribution of the number of clusters, and a principled way to report clustering uncertainty — none of which \(k\)-means or a dendrogram provides.

Other nonparametric objects#

The DP is one of a family. The Indian buffet process gives a nonparametric prior for latent-feature models — objects possessing an unbounded set of overlapping features rather than belonging to one cluster. Pólya trees and Gaussian processes are nonparametric priors on densities and functions. Survival and hazard functions get nonparametric treatments too. The unifying theme: put a prior on an infinite-dimensional object — a distribution, a function, a feature matrix — and let the data determine its complexity, with every downstream quantity carrying honest posterior uncertainty.

Hint