🔁 Gaussian Processes (GPs)

Gaussian Processes (GPs)#

A nonparametric Bayesian prior over functions that yields predictions with calibrated uncertainty.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The big idea#

A Gaussian process (GP) is a distribution over functions. Just as a Gaussian over numbers is fixed by a mean and variance, a GP over functions is fixed by a mean function and a covariance function (kernel). Rather than positing parameters (like neural-network weights) and fitting them, a GP says directly: here is the family of functions I believe in, with uncertainty around it.

Formal definition#

\[f(x) \sim \mathcal{GP}\big(m(x),\, k(x, x')\big),\]

with mean \(m(x) = \mathbb{E}[f(x)]\) and kernel \(k(x, x') = \operatorname{Cov}(f(x), f(x'))\). The defining property: any finite set of inputs has a joint multivariate Gaussian,

\[(f(x_1), \dots, f(x_n)) \sim \mathcal{N}(\mathbf{m}, K),\]

where \(K_{ij} = k(x_i, x_j)\) is built from the kernel.

Kernels encode your assumptions#

The kernel decides how correlated nearby inputs are:

RBF / squared-exponential — \(k(x, x') = \exp\!\big(-\lVert x - x' \rVert^2 / 2\ell^2\big)\) — smooth functions, with the length-scale \(\ell\) setting how fast correlation decays.
Linear — straight-line trends.
Periodic — repeating patterns.

Kernels compose (sum or product) to build structure like trend + seasonality.

Posterior inference#

Given training data \((X, y)\) and test inputs \(X_*\), the GP prior makes \(y\) and the test values \(f_*\) jointly Gaussian; conditioning on \(y\) yields a Gaussian posterior over \(f_*\) with both a mean prediction and a variance. So a GP returns a smooth curve and a calibrated confidence band — uncertainty for free.

Cost and trade-offs#

The catch is computation: conditioning inverts an \(n \times n\) covariance matrix at \(O(n^3)\) cost, so exact GPs suit small-to-medium data and need sparse / inducing-point approximations to scale. Kernel choice is critical — the wrong kernel gives poor predictions. In scikit-learn, GaussianProcessRegressor implements this.

GPs and neural networks#

A GP places uncertainty over functions; a Bayesian neural network places it over weights. The two meet at a famous limit: an infinitely wide neural network with random weights converges to a Gaussian process.

Theme: Bayesian Inference · All terminology

Hint

Mind map — connected ideas

Bayesian Neural Networks (BNNs) · Bayesian Time Series · Variational Inference (VI) · Posterior · Bayesian Inference.

Hint

More in Bayesian Inference

Bayes’ Theorem · Bayesian Correction · Bayesian Decision Theory (BDT) · Bayesian Inference. · Bayesian Neural Networks (BNNs) · Binomial Likelihood · Marginal Likelihood (also called The Model Evidence or Integrated Likelihood) · MCMC (Markov Chain Monte Carlo) · Parameter(s) of Interest · Posterior · Posterior belief · Posterior Probability · Posterior probability of uplift · Prior Belief (or Prior Probability)