Best Linear Predictor of a Stationary Process#

Stage 4 · 🎯 Prediction & the Sample ACF / PACF · Lesson 09 of 18 · intermediate

◀ Previous · Understanding ACFs via Difference Equations for AR(p) and ARMA(p, q) · Next · Sample ACF and Sample PACF ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The prediction problem#

Given a stationary series observed up to now, the best linear predictor of the next value is the linear combination of past observations \(\hat{x}_{n+1} = \sum_{k=1}^{n} a_k\, x_{n+1-k}\) that minimises the mean-squared error. “Best” here means within linear rules; for a Gaussian process it is the best predictor of any kind.

The projection principle#

The solution is geometric: treat random variables as vectors with inner product \(\langle X, Y\rangle = \mathbb{E}[XY]\), and the best predictor is the orthogonal projection of the target onto the span of the predictors. Optimality is characterised by the orthogonality principle — the prediction error is uncorrelated with every predictor:

\[\mathbb{E}\big[(x_{n+1} - \hat{x}_{n+1})\, x_{n+1-k}\big] = 0, \qquad k = 1, \dots, n.\]

Writing these conditions out gives the prediction equations, a linear system \(\Gamma_n \mathbf{a} = \gamma_n\) in the autocovariances — the same Toeplitz system as Yule–Walker.

Solving efficiently#

Solving that system afresh at each order is wasteful. The Durbin–Levinson recursion builds the order-\(h\) predictor from the order-\((h-1)\) one without inverting a matrix, updating the coefficients and the error variance in place — cheap and numerically stable.

From predictor to PACF#

The recursion delivers a bonus. The last coefficient \(\phi_{hh}\) of the best order-\(h\) predictor is the partial autocorrelation at lag \(h\) — the correlation between \(x_t\) and \(x_{t-h}\) after removing the linear influence of the intervening values. For an AR(p) these coefficients vanish for \(h > p\), so the PACF cuts off at lag p — the property that makes it the tool for reading autoregressive order.

Hint