Forward Selection with Nested Models and Deviance Tests#

Stage 6 · 🎯 Classification & Logistic Regression · Lesson 42 of 56 · advanced

◀ Previous · Complete and Quasi-Complete Separation in Logistic Regression · Next · Interpreting and Assessing a Forward-Selection Logistic Regression Model for College Student Retention ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Selecting features by deviance#

The deviance — that \(-2\) log-likelihood measure of misfit — does more than grade a single model. Its real power is in comparing models, and it is the natural tool for feature selection in logistic regression, playing the role that the F-test and residual sum of squares play in linear regression.

Nested models#

The comparison requires the models to be nested: one model’s predictors must be a subset of the other’s, so the smaller (reduced) model is a special case of the larger (full) one. Adding predictors can only lower the deviance (improve the fit on the training data), so the full model always has deviance less than or equal to the reduced model. The question is whether that improvement is real or just the inevitable reward of extra parameters.

The deviance test#

The answer is the likelihood-ratio (deviance) test. The difference in deviance between the reduced and full models is itself a statistic that, under the hypothesis that the extra features add nothing, follows a chi-square distribution — with degrees of freedom equal to the number of added parameters:

\[\Delta D = D_{\text{reduced}} - D_{\text{full}} \sim \chi^2_{\,k}.\]

A large deviance drop is unlikely by chance, so a significant test means the added features genuinely improve the model; a small, non-significant drop means they can be dropped. This procedure is often called an analysis of deviance, the logistic cousin of analysis of variance.

Forward selection, revisited#

This gives forward selection a principled entry rule for logistic models. Start from the null model and, at each step, consider adding each remaining feature; admit the one whose deviance drop is largest and significant by the chi-square test. Stop when no candidate produces a significant improvement. It is the same greedy search as before, now driven by deviance rather than an F-statistic — and with the same caution that data-driven selection inflates significance, so the final model earns its keep only on held-out data. The next lesson works a full example on real data.

Hint