Density regression#

Part 5 · Stage 16 · ♾️ Mixtures & Nonparametric Bayes · Lesson 144 of 144 · advanced

◀ Previous · Hierarchical dependence · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The whole distribution as a function of x#

The final synthesis. Ordinary regression models how the mean of \(y\) changes with predictors; even flexible regressions (splines, GPs) usually still assume the shape of the conditional distribution is fixed. Density regression removes that last assumption, letting the entire conditional distribution \(p(y \mid x)\) — its variance, skewness, modality, everything — vary with \(x\).

Why the mean is not enough#

Sometimes the interesting effect is not on the average. A treatment might not shift the mean response but split it into responders and non-responders — a change from one mode to two. Income dispersion may widen with education while the mean barely moves. Any model that reports only \(\mathrm{E}[y \mid x]\) is blind to these, and they are often the substantive finding.

Dependent Dirichlet processes#

The tool is a DP mixture whose ingredients depend on \(x\) — a dependent Dirichlet process (MacEachern). The mixture that generates \(y\) has weights and/or component parameters that are functions of the predictors, so the whole conditional density is free to change smoothly across the covariate space:

\[p(y \mid x) = \sum_{k=1}^{\infty} \pi_k(x) \, f\bigl(y \mid \theta_k(x)\bigr),\]

with \(\pi_k(x)\) and \(\theta_k(x)\) varying with \(x\) — for instance weights from a covariate-dependent stick-breaking, or component means that are themselves regressions or Gaussian processes in \(x\).

import pymc as pm
# covariate-dependent mixture: component means are regressions in x
K = 15
with pm.Model():
    beta = pm.Beta("beta", 1, pm.Gamma("alpha", 1, 1), shape=K)
    w = pm.Deterministic("w", beta * pm.math.concatenate(
        [[1.0], pm.math.cumprod(1 - beta)[:-1]]))
    coef = pm.Normal("coef", 0, 1, shape=(K, X.shape[1]))    # each component a regression
    mus = X @ coef.T                                          # component means depend on x
    # observation drawn from the x-dependent mixture (weights could depend on x too)
    pm.NormalMixture("y", w=w, mu=mus, sigma=pm.HalfNormal("s", 1, shape=K), observed=y)

The synthesis, and the close#

Density regression unites the whole book. It is a regression (Part IV), made flexible by a mixture (this stage), rendered nonparametric by a Dirichlet process (the last lessons), often with Gaussian-process or spline components (Part V) and hierarchical sharing across groups (throughout) — and fitted, checked and compared by the computational and model-checking machinery of Parts II and III. It is the most general regression in this course: predictors may reshape the response distribution in any smooth way, with full posterior uncertainty over the entire family of conditional densities. From Bayes’ rule for a single unknown to an infinite, covariate-indexed family of distributions — the arc of Bayesian data analysis, from its one idea to its fullest expression.

Hint