Logistic Regression: Modeling Binary Outcomes via Odds and Log-Odds#

Stage 6 · 🎯 Classification & Logistic Regression · Lesson 38 of 56 · advanced

◀ Previous · How Shapley Values Work · Next · Maximum Likelihood (MLE): Fitting a Distribution to Observed Data ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

When the outcome is yes or no#

Regression so far has predicted a number — a fare, a price. But many outcomes are binary: will a customer churn or not? will a student return next year? Ordinary linear regression fails here, because a straight line runs off to \(\pm\infty\) and would predict “probabilities” below 0 or above 1. Logistic regression is the standard model for a yes/no outcome, and it works by predicting a probability instead.

Odds and log-odds#

The trick is to transform the probability so a linear model fits. Start with the odds — the ratio of the probability of the event to its complement, \(p / (1 - p)\) — which stretches \([0, 1]\) out to \([0, \infty)\). Then take the logarithm, giving the log-odds or logit, which spans all real numbers. Logistic regression makes this linear in the features:

\[\ln\!\left(\frac{p}{1 - p}\right) = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p.\]

The model is an ordinary linear equation — just on the log-odds scale rather than the probability scale.

The logistic curve#

To get a probability back, invert the transform. Solving for \(p\) gives the logistic (sigmoid) function:

\[p = \frac{1}{1 + e^{-z}}, \qquad z = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p.\]

This S-shaped curve takes the linear combination \(z\) — any real number — and squashes it smoothly into a valid probability between 0 and 1. Large positive \(z\) gives \(p\) near 1, large negative near 0, and \(z = 0\) gives \(p = 0.5\).

Reading coefficients#

The coefficients read on the odds scale. A one-unit rise in \(x_j\) adds \(\beta_j\) to the log-odds, which multiplies the odds by \(e^{\beta_j}\) — the odds ratio. So \(e^{\beta_j} > 1\) means the feature raises the odds of the event, \(< 1\) lowers them. In Python it is LogisticRegression in scikit-learn, or Logit in statsmodels for the full coefficient table. Unlike least squares, its coefficients have no closed form — they are found by maximum likelihood, the subject of the next lesson.

Hint