Gradient Descent on m Training Examples#

Stage 4 · ⚙️ Backprop & Vectorization · Lesson 14 of 17 · intermediate

◀ Previous · Logistic Regression Gradient Descent · Next · Vectorization in Logistic Regression ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Averaging the gradient#

A single example gives a noisy gradient; training uses the whole set. The cost is the average loss, so — because differentiation is linear — its gradient is the average of the per-example gradients:

\[J(\mathbf{w}, b) = \frac{1}{m}\sum_{i=1}^{m} \mathcal{L}(a^{(i)}, y^{(i)}), \qquad \frac{\partial J}{\partial w_j} = \frac{1}{m}\sum_{i=1}^{m} x_j^{(i)}\,\mathrm{d}z^{(i)}, \qquad \frac{\partial J}{\partial b} = \frac{1}{m}\sum_{i=1}^{m} \mathrm{d}z^{(i)}.\]

The explicit algorithm#

Written out naively, one gradient-descent step accumulates over the examples, then divides by \(m\):

J = 0; dw1 = 0; dw2 = 0; db = 0
for i in range(m):                     # loop over the m examples
    z = w1 * x1[i] + w2 * x2[i] + b
    a = sigmoid(z)
    J += -(y[i] * log(a) + (1 - y[i]) * log(1 - a))
    dz = a - y[i]
    dw1 += x1[i] * dz                  # one line per feature ...
    dw2 += x2[i] * dz
    db += dz
J /= m; dw1 /= m; dw2 /= m; db /= m
w1 -= alpha * dw1; w2 -= alpha * dw2; b -= alpha * db

Two loops, and slow#

Look closely and there are two nested loops: the visible one over the \(m\) examples, and a hidden one over the \(n\) features — with \(n\) weights you would need \(\mathrm{d}w_1, \mathrm{d}w_2, \dots, \mathrm{d}w_n\). Explicit Python loops like these are badly suited to parallel hardware and crawl on large datasets. Removing them — vectorization — is the subject of the next lesson.

Hint