Cloud Inference#
Running model predictions on managed cloud infrastructure.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Cloud inference is running a trained model on cloud infrastructure to make predictions on new data. “Inference” means using a trained model rather than training one, and “cloud” means it runs on AWS, GCP or Azure rather than on local or on-device hardware.
How it works#
Three stages. Train a model (locally or in the cloud) and save its weights. Deploy it to a cloud service (SageMaker, Vertex AI, Azure ML) behind an API endpoint (REST or gRPC). Then infer: the client sends input, the service loads the model, runs a forward pass and returns the prediction.
Benefits and challenges#
The upsides are scalability (autoscale to millions of requests), low maintenance (the provider runs servers and GPUs), flexibility (serve and version many models), and accessibility (any device can call the API). The costs are latency (a network round trip), OpEx per prediction at scale, privacy obligations when sensitive data leaves the device, and the need for monitoring to keep uptime and fairness in check.
Cloud vs edge#
The contrast is with edge / on-device inference. Cloud runs on powerful servers (GPUs, TPUs) and is easy to update but pays a network-latency cost; edge runs locally on a phone or IoT device with low latency but limited compute and harder remote updates. Image-tagging APIs, LLM chat services and fraud-scoring endpoints are all cloud inference.
Theme: MLOps, Serving & Monitoring · All terminology
Hint
Mind map — connected ideas
Cloud Inference with Big Payloads · AWS SageMaker Endpoints · OpenAI API (ML API) · AWS SageMaker · Vertex AI · Quantization
Hint
More in MLOps, Serving & Monitoring
AWS SageMaker Endpoints · Caching · Cloud Inference with Big Payloads · Compute budgets · Continuous Retraining · Feature Values · Guardrails (in ML & Data Systems) · Inference Cost (Inference $) · Latency Guardrails · Manual review minutes · Model KPIs (Key Performance Indicators) · Model Stability · Monitoring Pipelines · Ops Health Dashboard
See also
Source article Adapted (context, re-expressed) in our own words from: Cloud Inference (insightful-data-lab.com).