🏋️ Model Distillation (Knowledge Distillation)

Model Distillation (Knowledge Distillation)#

Training a small student model to mimic a larger teacher for efficiency.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Model distillation (knowledge distillation) is a compression technique: a large, accurate teacher model transfers its knowledge to a smaller, faster student. The aim is a lighter model that keeps most of the teacher’s accuracy while being cheap enough to deploy on phones and edge devices.

How it works#

Train the big teacher (a large transformer, say), then have it produce soft targets — full probability distributions like [0.7 cat, 0.2 dog, 0.1 rabbit] rather than a bare hard label. The student trains to mimic those soft outputs, usually under a loss that blends a distillation term (student vs teacher) with a supervised term (student vs true labels).

The objective#

\[L = \alpha \, L_{\text{hard}}(y, p_s) \; + \; (1 - \alpha)\, L_{\text{soft}}(p_t, p_s, T),\]

where \(y\) are true labels, \(p_t\) and \(p_s\) the teacher and student probabilities, \(T\) a temperature that softens the teacher’s distribution to expose its “dark knowledge”, and \(\alpha\) the balance between the two terms.

Why, examples, and costs#

It buys efficiency (faster, cheaper inference), deployability (edge devices), and even better generalisation from the teacher’s soft probabilities. DistilBERT is ~40% smaller and ~60% faster than BERT at ~97% of its performance; ResNet-50 distills into ResNet-18 at similar accuracy. The catches: the student cannot capture everything, \(T\) and \(\alpha\) need tuning, and you still pay to train the teacher first.

Theme: Model Training & Optimization · All terminology

Hint

Mind map — connected ideas

Quantization · Frozen Encoder · Embedding · Autoencoder · Early Stopping · Epochs

Hint

More in Model Training & Optimization

Active Learning · Binary Cross-Entropy (BCE) · Deep Ensembles · Early Stopping · Ensemble · Epochs · FLOPs · Full Annotation · Hyperparameter · Label Noise · Log-Odds · Logit Space · Logits · Loss Functions