🏋️ Softmax Function

Softmax Function#

Converts a vector of scores into a probability distribution over classes.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

The softmax function generalizes the sigmoid to many classes. It takes a vector of K raw scores (logits), exponentiates each and normalizes by their sum, producing K probabilities that each lie in (0,1) and sum to 1 — a full distribution over mutually exclusive classes:

\[\text{softmax}(z)_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}.\]

Its role#

It is the standard output layer for multi-class classification, trained with categorical cross-entropy. It amplifies the largest score toward 1 while dampening the rest — a soft “winner.” When \(K=2\) it reduces to the sigmoid.

Watch out#

Because the outputs are coupled (they must sum to 1), softmax assumes classes are mutually exclusive — for multi-label problems (independent classes) use per-class sigmoids instead. Its probabilities can also be poorly calibrated.

Theme: Model Training & Optimization · All terminology

Hint

Mind map — connected ideas

Sigmoid Function · Squashing Function · Log-Odds · Neural Networks · Classification Models · Binary Classification

Hint

More in Model Training & Optimization

Active Learning · Binary Cross-Entropy (BCE) · Deep Ensembles · Early Stopping · Ensemble · Epochs · FLOPs · Full Annotation · Hyperparameter · Label Noise · Log-Odds · Logit Space · Logits · Loss Functions