🏋️ Underflow

Underflow#

Numerical loss of precision when values become too small to represent.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Underflow happens when a computation produces a number too small to represent in the floating-point format — smaller than the tiniest positive value — so the computer rounds it to zero, destroying a real nonzero result. It is the small-magnitude counterpart of overflow.

Why ML hits it#

Machine learning multiplies many small probabilities — in Naive Bayes, HMMs, and likelihoods — and the product of hundreds of values below 1 quickly drops below the representable floor, collapsing to 0 and corrupting the result. Low-precision (float16) training underflows even sooner, showing up as vanishing gradients.

The fix#

Compute in log space. Because \(\log(a \cdot b) = \log(a) + \log(b)\), a fragile product of tiny probabilities becomes a stable sum of log-probabilities — the reason libraries use log-likelihoods and the LogSumExp trick, and why scikit-learn’s Naive Bayes works with logs internally.

Theme: Model Training & Optimization · All terminology

Hint

Mind map — connected ideas

Logits · Softmax Function · Log Loss (also called Logarithmic Loss or Cross-Entropy Loss) · Log-Odds · Quantization · Sigmoid Function

Hint

More in Model Training & Optimization

Active Learning · Binary Cross-Entropy (BCE) · Deep Ensembles · Early Stopping · Ensemble · Epochs · FLOPs · Full Annotation · Hyperparameter · Label Noise · Log-Odds · Logit Space · Logits · Loss Functions