Feature Importance in Linear Regression#

Stage 5 · 📈 Regression · Lesson 33 of 56 · intermediate

◀ Previous · Multiple Linear Regression · Next · Forward Selection: Definition and Core Idea ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Which features matter?#

Once a multiple regression is fitted, a natural question follows: which predictors matter most? Which features do the heavy lifting in explaining the outcome, and which barely register? The coefficients seem like the obvious answer — bigger coefficient, bigger effect — but read naively, they mislead.

The units trap#

The problem is units. A coefficient’s size depends on the scale of its feature. Suppose fare rises by one dollar per mile and by one cent per second travelled; the mile coefficient (1.0) dwarfs the second coefficient (0.01), yet that reflects the units, not the importance — a mile is simply a bigger step than a second. Comparing raw coefficients across features measured in different units is comparing apples to oranges.

Standardized coefficients#

The fix is to put every feature on the same scale before comparing. Standardising each predictor — subtracting its mean and dividing by its standard deviation, so all are measured in standard-deviation units — yields standardised coefficients (often called beta coefficients). Now each answers a comparable question: how much does \(y\) move for a one-standard-deviation change in this feature? Their magnitudes can be ranked, giving a genuine measure of importance. Statistical significance (the t-statistic and p-value from statsmodels) tells a complementary story: whether an effect is distinguishable from zero at all.

Cautions#

Two caveats. Multicollinearity — predictors that are themselves strongly correlated — makes individual coefficients unstable and their importances hard to trust, since the model cannot cleanly separate overlapping features (recall the correlation lessons of Stage 2). And linear-model importance only captures linear effects; a feature with a strong curved relationship may look unimportant. For a model-agnostic alternative, permutation importance measures how much performance drops when a feature is shuffled — an idea that generalises to the trees and other models ahead. The next lessons turn to choosing which features to include at all.

Hint