Why Deep Learning is Taking Off#

Stage 1 · 🧠 Introduction to Deep Learning · Lesson 03 of 17 · beginner

◀ Previous · Supervised Learning and Neural Networks · Next · Geoffrey Hinton Interview ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

An old idea, newly working#

The mathematics of neural networks is decades old, so why the recent explosion? Not one breakthrough but a convergence — the raw ingredients finally reached the scale where deep networks decisively outperform the alternatives.

Scale drives performance#

Ng summarises it with a single picture: plot performance against the amount of labelled data. Traditional methods (logistic regression, SVMs) improve for a while, then plateau. Neural networks keep climbing, and bigger networks climb higher — small < medium < large. The pattern has two regimes: with little data, careful feature engineering and skill can matter more than model size, so the ordering blurs; with lots of data, a large network wins clearly.

The three drivers#

Three forces made that scale reachable. Data — a digitised world (phones, sensors, the web) produces the huge labelled datasets networks feed on. Computation — GPUs, faster hardware and distributed training make large models trainable in reasonable time. Algorithms — better design speeds learning; the switch from the sigmoid to the ReLU activation is the classic example, easing the vanishing-gradient problem so gradient descent converges much faster.

The virtuous cycle#

These drivers reinforce each other. Faster hardware and better algorithms shorten the idea → code → experiment loop, so researchers iterate more quickly; better models attract more users, who generate more data, which trains still-better models. That feedback loop is much of why progress has felt so fast — and why the fundamentals in this course sit beneath so many modern systems.

Hint