Gradient Descent in Logistic Regression#

Stage 3 · 📉 Derivatives & the Computation Graph · Lesson 08 of 17 · intermediate

◀ Previous · Logistic Regression – Loss Function and Cost Function · Next · Derivatives ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

The optimization problem#

With a cost \(J(\mathbf{w}, b)\) in hand, learning becomes a search: find the \(\mathbf{w}, b\) that make \(J\) smallest. The cross-entropy cost for logistic regression is convex — a single bowl-shaped surface with one global minimum and no misleading local dips — which is what makes the search reliable.

Rolling downhill#

Gradient descent finds that minimum by repeatedly stepping downhill. At the current point the gradient — the partial derivatives \(\partial J / \partial \mathbf{w}\) and \(\partial J / \partial b\) — points in the direction of steepest increase; moving the opposite way decreases the cost. Start anywhere (for a convex cost, even all-zeros works) and repeat.

The update rule#

Each iteration nudges the parameters against the gradient:

\[\mathbf{w} := \mathbf{w} - \alpha\,\frac{\partial J}{\partial \mathbf{w}}, \qquad b := b - \alpha\,\frac{\partial J}{\partial b}.\]

In code the derivatives are conventionally named dw and db, so the step reads w -= alpha * dw and b -= alpha * db. Repeat until the cost stops decreasing.

The learning rate#

The step size \(\alpha\) is the learning rate. Too small and training crawls, needing many iterations; too large and the steps overshoot the minimum and may diverge. Choosing \(\alpha\) well — and computing those derivatives efficiently — is what the rest of this stage is about, starting with the calculus itself.

Hint