A shared kitchen with a few very expensive ovens. Instead of baking one pizza at a time, the chef keeps every oven full, sliding a new pizza in the moment one comes out, and sends each order to the oven with room and the right ingredients already warm.
Problem Statement
Design the platform that serves large language models to many internal products and external API customers: requests come in with a prompt, run on GPU servers, and return generated tokens. The goal is high throughput and low latency from very expensive, very scarce hardware, while isolating customers from each other.
Functional Requirements
Serve several models (sizes and versions) behind one API, with streaming and non-streaming responses.
Route requests to healthy replicas of the requested model; roll out new model versions safely.
Per-tenant quotas (requests and tokens per minute) and priorities.
Usage metering for billing.
Out of scope: model training and fine-tuning pipelines.
Non-Functional Requirements
Latency: time to first token (TTFT) and time per output token (TPOT) targets per tier.
Throughput / cost: maximise tokens per second per GPU.
Availability: survive GPU/node failures and zone outages; no single model version outage takes the API down.
Fairness: one tenant's burst mustn't starve others.
Rough numbers
a 70B-parameter model in 16-bit weights ~ 140 GB -> spread across 2-4+ GPUs (tensor parallel)
per request: prefill (read the prompt, compute-bound) then decode (1 token per step, memory-bound)
KV cache per sequence grows with context length -> GPU memory, not compute, often caps batch size
Core Entities
Model and ModelVersion (weights location, hardware needs, quantisation).
Replica: a set of GPUs serving one model version; state (loading/ready/draining), load.
Tenant with quotas and priority; Request with token counts and timings.
API Design
POST /v1/generate
{ "model": "chat-large", "messages": [...], "max_tokens": 800, "stream": true }
headers: Authorization (tenant), Idempotency-Key (optional)
-> SSE stream of tokens, then usage {input_tokens, output_tokens}
-> 429 with Retry-After when the tenant's quota is exhausted
-> 503 when capacity is exhausted (client may retry with backoff)
High-Level Design (HLD)
Route each request to the replica with the most free capacity (and a warm prefix cache); canary new versions.
API gateway: auth, per-tenant token-bucket limits (on requests and on tokens), request validation.
Router / scheduler: picks a replica of the right model using live load (queue depth, free KV-cache memory) and prefix affinity (send the same system prompt to replicas that already cache it).
Model servers (vLLM, TensorRT-LLM, TGI-style engines) run continuous batching on GPUs.
Autoscaler + model registry: scale replicas on queue depth/latency, preload weights; canary new versions.
Metering: usage events to a stream for billing.
Low-Level Design (LLD)
Sequences join and leave the batch every step, so the GPU never waits for the slowest request.
Continuous batching and the KV cache
Static batching: wait for N requests, run them together until ALL finish
-> short answers wait for the longest one; GPU idles as sequences end
Continuous (in-flight) batching: every decode step, finished sequences leave
the batch and waiting ones join
step 1: [A B C D] step 2: [A B C D] step 3: [A C D E] (B done, E joined)
KV cache: the attention keys/values of every token so far, per sequence.
Paged KV cache (PagedAttention) allocates it in fixed blocks, like OS pages,
so memory isn't wasted on max-length reservations and prefixes can be shared.
Routing
Load-aware: prefer the replica with the most free KV-cache blocks and shortest queue ("least outstanding tokens"), not round robin: requests differ hugely in cost.
Prefix-aware: hash the first N tokens (system prompt) and prefer replicas that have that prefix cached, cutting prefill work.
Priority queues per tier: interactive traffic ahead of batch/offline jobs; batch jobs fill idle capacity.
Scaling and rollout
Loading weights takes minutes, so autoscaling is predictive (time-of-day) plus reactive (queue depth, TTFT), with a warm pool of ready replicas.
Canary a new model version to a small traffic share; compare quality metrics and latency before shifting more.
Separating prefill and decode onto different GPU pools ("disaggregated serving") is an advanced option when prompts are long.
Follow-up Questions
Why is round-robin load balancing bad here?
Request cost varies by orders of magnitude (a 100-token prompt vs a 100K-token document). Round robin ignores that and piles long jobs onto one replica; route on outstanding tokens and free KV memory instead.
How do you rate-limit fairly?
Limit both requests per minute and tokens per minute per tenant (estimated input + max output on admission, corrected with actual usage after), with priority tiers so batch traffic yields to interactive traffic.
How does quantisation help?
Storing weights in 8 or 4 bits shrinks memory, letting a model fit on fewer GPUs and leaving more memory for KV cache (bigger batches), usually at a small quality cost that must be measured.