🏋️ FLOPs

FLOPs#

Floating-point operations — a measure of a model’s compute cost.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

FLOPs — floating-point operations — count the mathematical operations (adds, multiplies, divides) a model or algorithm performs. In ML, FLOPs are a standard proxy for computational complexity and for training/inference cost — they measure workload, not memory access or I/O.

The units#

One operation is a FLOP, and the scale climbs fast: MFLOPs (millions), GFLOPs (billions), TFLOPs (trillions), PFLOPs (quadrillions). A single large matrix multiplication already involves enormous counts.

Counting them#

Multiplying an \(m \times n\) matrix by an \(n \times p\) matrix costs about \(2 \, m \, n \, p\) FLOPs — so two \(1000 \times 1000\) matrices run to roughly 2 billion. In deep learning, training FLOPs scale with dataset size × model size × epochs, while inference FLOPs per forward pass drive deployment latency. For comparison, ResNet-50 is ~4 GFLOPs per image, BERT-base ~22 GFLOPs per sequence, and GPT-3 took an estimated \(3 \times 10^{23}\) FLOPs to train.

FLOPs vs FLOPS#

Two letters, two meanings. FLOPs is a count of operations (the workload); FLOPS is floating-point operations per second, the speed of hardware — an NVIDIA A100 delivers ~312 TFLOPS on its tensor cores. FLOPs is the job; FLOPS is how fast the worker does it. Because training cost scales almost linearly with FLOPs, the count is a key lever in the efficiency-versus-accuracy trade-off.

Theme: Model Training & Optimization · All terminology

Hint

Mind map — connected ideas

Model Weights · Quantization · Model Distillation (Knowledge Distillation) · Cloud Inference · OpEx · Neural Networks

Hint

More in Model Training & Optimization

Active Learning · Binary Cross-Entropy (BCE) · Deep Ensembles · Early Stopping · Ensemble · Epochs · Full Annotation · Hyperparameter · Label Noise · Log-Odds · Logit Space · Logits · Loss Functions · Model Distillation (Knowledge Distillation)