Latency SLAs - real-time (<100ms p99), interactive (<1s), batch (minutes-hours)?

Online inference p99 < 100ms for tabular, < 500ms for vision, < 2s for LLM completion. Batch inference: minutes to hours, throughput-optimized. Cold-start a new replica: < 60s.

Sitelet https://gitgood.dev/system-design/ml-serving

gitGood.dev

Distributed Systems

Design an ML Model Serving Platform (TorchServe / Triton)

Hard Premium

Online vs batch inference, GPU utilization tricks, autoscaling for spiky load, A/B testing models, and the feature store that decouples training from serving.

The problem

Design a platform that serves ML model predictions to internal product teams. Hundreds of models in production at any time, each with different latency, throughput, and accuracy SLAs. Some run on GPUs (LLMs, image models), others on CPU (tabular). Traffic is spiky: a recommendation model might idle at 100 req/sec and spike to 50K during a product launch. Teams want to A/B test model versions safely and roll back instantly when a new model regresses.

This is the canonical "ML platform infra" problem. Strong candidates separate online vs batch from the start, design for GPU utilization (the cost driver), and own the model registry + rollout story. Excellent candidates discuss the feature-store split between training and serving and explain why training-serving skew is the most common production ML failure.

Clarifying questions

Asking these before diving into a solution is the difference between a "hire" and a "no signal" rating. Pick the questions whose answers would change your design.

→What's the model mix - LLMs, computer vision, tabular, recommendation? Each has very different infra needs.
→Latency SLAs - real-time (<100ms p99), interactive (<1s), batch (minutes-hours)?
→Throughput - hundreds of QPS or millions?
→Are inputs simple (a feature vector) or rich (long text, images, video)?
→Stateful inference (chat sessions with KV cache, ranking with personalization) or stateless (single request, single response)?
→How often do models update - hourly, daily, weekly? Hot-swap or full redeploy?
→Multi-tenancy - one model per pod, or many models packed onto shared GPUs?
→Training-serving feature parity required, or are some features online-only?

Requirements

Functional requirements

·Deploy a model: register a versioned artifact and route a portion of traffic to it
·Serve predictions: low-latency synchronous API + async batch API
·Route traffic between model versions (A/B, shadow, canary, rollback)
·Hot-swap a model with no in-flight request loss
·Per-model and per-version metrics (latency, throughput, accuracy proxies)
·Feature store integration: serving features must match training features
·Capacity isolation: a hot tenant can't starve other tenants

Non-functional requirements

Scale: 1000 active models. 100K aggregate QPS at peak. 1000 GPUs (mixed A100, L40, T4). 10x traffic variance day-vs-night. New model deploy: 30 minutes from artifact upload to live traffic.
Latency: Online inference p99 < 100ms for tabular, < 500ms for vision, < 2s for LLM completion. Batch inference: minutes to hours, throughput-optimized. Cold-start a new replica: < 60s.
Availability: 99.95% per model. Platform-level: 99.99% (a single model failure must not cascade). Rollback to prior version: < 60s.
Consistency: Best-effort. Stateless inference is naturally consistent. Stateful (chat KV cache, session-affine ranking) requires sticky routing. Feature freshness is an SLA, not strict consistency.

The full walkthrough covers

Architecture components

·Model registry
·Deployment manager
·Inference router
·Inference server (TorchServe / Triton / vLLM)
·Batch inference scheduler
·Feature store (online)
·Feature store (offline)
·Traffic controller
·Observability stack
·Autoscaler

Deep dives

Online vs batch inference: the two modes and where they meet
GPU utilization: batching, packing, KV cache, and the cost ceiling
Autoscaling for variable load: scale by what, exactly?
A/B, shadow, canary, rollback: the safe deploy story
Feature store: training-serving parity and the most common production ML failure
Observability for ML: what to monitor when the model itself can be wrong

What's in the full walkthrough

Capacity estimation · architecture with all 10 components explained · 6 deep dives · trade-off analysis · 8 common follow-up questions

This system design walkthrough is part of Premium

Unlock it with a subscription.

Start Your 7-Day Trial

Try everything free for 7 days. Cancel anytime.

Every system design walkthrough - capacity estimates, deep dives and trade-offs
Full-length AI mock interviews - 3 to 10 questions with follow-ups, up to 3 a day

View Plans

7-day trial, then $8/mo - or $48/yr ($4/mo, save 50%). Cancel anytime.

Companies that test this topic