💡 Logistic Regression

Logistic Regression#

A linear model mapping features to a probability via the logistic function.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Logistic regression is a linear model for binary classification that predicts the probability of the positive class. Despite regression in the name it classifies: it outputs a probability, then a threshold (usually 0.5) assigns the label.

The sigmoid and log-odds#

It passes a linear combination through the sigmoid, squashing any real number into \((0, 1)\):

\[\sigma(z) = \frac{1}{1 + e^{-z}}, \qquad z = \theta_0 + \boldsymbol{\theta}^\top \mathbf{x}.\]

Equivalently, the log-odds (logit) — the natural log of the odds \(p/(1-p)\) — is linear in the features, which is what makes the coefficients interpretable:

\[\log \frac{p}{1 - p} = \theta_0 + \boldsymbol{\theta}^\top \mathbf{x}.\]

Fitting by maximum likelihood#

Coefficients are chosen to maximize the likelihood of the observed labels — equivalently, to minimize the log loss (negative log-likelihood), a convex objective solved by gradient-based methods:

\[\mathcal{L} = -\sum_i \big[\, y_i \log \hat{p}_i + (1 - y_i)\log(1 - \hat{p}_i) \,\big].\]

Assumptions and multiclass#

It assumes a linear log-odds relationship, few extreme outliers, and enough data; it extends to several classes via one-vs-rest or multinomial (softmax). Simple, fast and interpretable, it is a workhorse for spam, fraud and medical diagnosis.

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
proba = clf.predict_proba(X_test)[:, 1]   # P(class 1)

Theme: AI & ML Concepts · All terminology

Hint

Mind map — connected ideas

Linear Models · Support Vector Machines (SVMs) · Neural Networks · Decision Trees · Multiclass AUROC · Discriminatory Power

Hint

More in AI & ML Concepts

AI (Artificial Intelligence) · Classification Models · Computer Vision (CV) · Decision Trees · Linear Models · LLMs (Large Language Models) · Machine Learning (ML) · Medical AI · Natural Language Processing (NLP) · Neural Networks · Regression Models · Support Vector Machines (SVMs) · Target Variable