🏋️ Loss Functions

Loss Functions#

Objectives quantifying prediction error that training seeks to minimise.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A loss function (cost or objective) measures how wrong a model’s predictions are on the training data — a single number the model minimizes. Lower loss means predictions closer to targets; it is the signal that gradient descent follows to update parameters.

Matching loss to task#

The loss encodes what “wrong” means — regression uses mean squared error or MAE; binary classification uses binary cross-entropy; multi-class uses categorical cross-entropy. Squared error can’t tell a bad classification from a disastrous one, which is why classification uses cross-entropy.

Why it matters#

The loss defines what the model actually optimizes, so a mismatched loss silently optimizes the wrong thing (MSE on a sigmoid gives a non-convex surface). A good loss is differentiable, fits the task, and aligns with the real objective.

Theme: Model Training & Optimization · All terminology

Hint

Mind map — connected ideas

Binary Cross-Entropy (BCE) · Logit Space · Mean Squared Error (MSE) · Sigmoid Function · Softmax Function · Neural Networks

Hint

More in Model Training & Optimization

Active Learning · Binary Cross-Entropy (BCE) · Deep Ensembles · Early Stopping · Ensemble · Epochs · FLOPs · Full Annotation · Hyperparameter · Label Noise · Log-Odds · Logit Space · Logits · Model Distillation (Knowledge Distillation)