⚙️ Compute budgets

Compute budgets#

Limits on the compute resources a workload may consume.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A compute budget is the pool of computational resources — GPU / TPU hours, FLOPs, and the dollars behind them — allocated to an ML system’s training and serving. It caps how big a model you can train and how much traffic you can serve.

The two halves#

Training is a one-time cost that grows with model and data size (its FLOPs approximated by the 6ND rule — roughly 6 × parameters × tokens), while inference is an ongoing cost (about 2N FLOPs per forward pass) that scales with usage:

\[C_{\text{train}} \approx 6ND, \qquad C_{\text{inf}} \approx 2N \ \text{FLOPs per pass}.\]

In production, inference usually claims the majority of the budget.

Managing it#

Teams set budgets and alerts, model optimistic / expected / pessimistic scenarios, and track unit economics like cost per prediction and GPU utilization. Hidden drains — idle instances, failed runs, oversized experiments — routinely add 20–40% over the planned figure.

Theme: MLOps, Serving & Monitoring · All terminology

Hint

Mind map — connected ideas

Inference Cost (Inference $) · TPU Clusters · Quantization · Caching · Cloud Inference · Latency Guardrails

Hint

More in MLOps, Serving & Monitoring

AWS SageMaker Endpoints · Caching · Cloud Inference · Cloud Inference with Big Payloads · Continuous Retraining · Feature Values · Guardrails (in ML & Data Systems) · Inference Cost (Inference $) · Latency Guardrails · Manual review minutes · Model KPIs (Key Performance Indicators) · Model Stability · Monitoring Pipelines · Ops Health Dashboard