Model serving is everything that turns a trained artifact into something other systems can call reliably: how the model is loaded, where it runs relative to your application, whether callers wait for the answer, how new versions are rolled out, and how the expensive hardware is shared. Interviewers probe it because "put the model behind an API" hides the real decisions: the same model behaves very differently as an embedded library, a synchronous microservice or an async job worker, and each choice fixes your latency profile, failure modes and cost. For a Backend or AI engineer the goal is to pick an architecture from the workload (request rate, latency budget, payload size, burstiness), and to explain what happens at the edges: a cold start, a traffic spike, a bad model version. Related pages: Model Serving Frameworks, Batching & Dynamic Batching and Multi-Model & Multi-Tenant Serving.
A model serving architecture is the overall design for how a trained AI model gets connected to the rest of the world so it can actually be used — where it physically runs, whether the app that calls it has to wait for an answer or gets notified later, how many requests it can handle at once, and what happens when a new version of the model replaces an old one. It is the plumbing between "we trained a model" and "our product can use it," and different designs suit different needs the way a restaurant might choose a walk-up counter, a reservation system, or a delivery service depending on how customers actually show up.
Think of a large hospital deciding how to route patients. Some cases need a doctor to see them immediately in the emergency room (synchronous, low latency). Others can be scheduled for a routine check-up next week (batch processing). And some get triaged into a queue where a nurse calls them back once a specialist is free (asynchronous). The hospital doesn't use one approach for everyone — it picks the routing method based on how urgent the case is, exactly like choosing a model serving architecture based on how quickly an answer is actually needed.
MODE TRIGGER LATENCY GOAL TYPICAL SHAPE EXAMPLE
online a user/service request ms to a few seconds request -> model server -> response fraud check at checkout
batch schedule / data drop minutes to hours job reads a table, scores all rows, nightly churn scores,
writes results to a store embedding a whole corpus
streaming events on a stream sub-second per event consumer (Kafka/Flink) -> model -> real-time anomaly flags
(two meanings below) sink topic / DB
"Streaming" is used for two different things:
(a) stream processing: an event stream is scored continuously, no caller waits
(b) response streaming: one request, but the answer is sent in pieces (LLM tokens over SSE/gRPC stream)
-> lowers PERCEIVED latency (time to first token), not total computeOnline inference must meet a latency SLO for each request, so you pay for idle capacity and headroom: the fleet is sized for peak, not average. Batch inference has no caller waiting, so it runs at maximum throughput on cheap or spot capacity and can use very large batches, but results are stale by the schedule interval. Many products use both: precompute what you can offline (embeddings, candidate lists, user scores) and run only the cheap final step online, which is the classic recommendation-system split. Choosing batch when freshness is not needed is often the single largest cost saving available.
The trap is treating "real time" as binary. Ask what staleness the product tolerates: if a fraud score computed every 5 minutes is acceptable, a streaming or micro-batch job is far simpler than a sub-100 ms online service. For LLM apps, response streaming (see Streaming Responses) changes the SLO to time-to-first-token plus inter-token latency, and the connection stays open for many seconds, which affects load balancers and timeouts.
EMBEDDED (library in the app process) MODEL-AS-A-SERVICE (separate deployment)
[ App process ] [ App ] --network--> [ Model server ] (GPU pool)
[ business logic + model ] scaled on CPU/RPS scaled on GPU/queue depth
no network hop; shared fate independent deploys, own hardware, shared by many apps
SYNC API ASYNC QUEUE
client --POST--> server --model--> 200 client --POST--> API --enqueue--> broker --> worker(s)
client holds the connection client <-- 202 + job_id; polls / webhook / push result
fits: results in < a few seconds fits: long jobs, bursty load, retries, batching for costEmbedded serving suits small models (a gradient-boosted tree, a tiny classifier, an on-device model) where a network hop would cost more than inference. It removes a dependency, but ties the model's memory, dependencies and release cycle to the app, and it cannot use a shared GPU. Model-as-a-service separates scaling and ownership: GPU nodes are expensive and slow to start, so you scale them differently from stateless web pods, batch requests from many callers together, and upgrade models without redeploying every client. The price is a network hop (typically about a millisecond or less inside a datacenter, but with real tail risk), one more failure domain, and a need for timeouts, retries and circuit breakers (see Fault Tolerance & Circuit Breakers and Timeouts, Retries & Backoff).
Sync versus async is a separate axis. A synchronous API is simplest and fits interactive calls, but it couples caller latency to model latency, and under overload requests time out while the server keeps working on abandoned ones. An async queue absorbs bursts (the queue is the shock absorber), permits retries and priority lanes, and lets workers batch aggressively, at the cost of a job-state store, polling or callbacks, and delivery semantics: brokers generally give at-least-once delivery, so workers must be idempotent (see Idempotency & Exactly-Once Semantics and Synchronous vs Asynchronous Processing). Queues also do not create capacity: if arrivals exceed service rate forever, the queue and the wait simply grow.
Time until a NEW replica can serve its first request (ILLUSTRATIVE assumptions; measure your own):
schedule pod on a GPU node / provision node : seconds to minutes (longer if a new node must join)
pull container image (several GB with CUDA libs) : tens of seconds, unless cached on the node
load weights: 14 GB at ~1-2 GB/s effective : ~7-14 s (more if read over the network)
initialise runtime (CUDA context, allocate KV cache / memory pools)
warm-up: first requests trigger kernel selection / compilation / autotuning, so they are slow
-----------------------------------------------------------------------------------------------
total: often tens of seconds to several minutes <- during a spike, this is your real reaction time
Mitigations: min replicas > 0 | warm pool of pre-loaded idle replicas | pre-pulled images / node caches
weights on local NVMe or a fast shared volume | readiness probe that waits for warm-up
scale on queue depth EARLY (before saturation) | scale-to-zero only for latency-tolerant modelsCold start is a first-principles arithmetic problem: bytes to move divided by bandwidth, plus one-time initialisation. A multi-gigabyte model cannot be "started" like a small web pod. The consequences are architectural: autoscaling reacts in minutes, not seconds, so you need headroom and predictive or queue-based scaling; a readiness probe must not report ready until the model is loaded and one dummy request has been run, or the load balancer will send real traffic to a replica that will answer slowly and time out. In Kubernetes this is the split between startup and readiness probes (see Kubernetes Basics and Autoscaling for Inference Workloads).
A warm pool keeps extra replicas already loaded and idle (or in a paused state), paying for idle GPU time to buy fast scale-out. Scale-to-zero (offered by serverless inference platforms such as Knative-based ones; check current provider behaviour) saves money for rarely-used models but makes the first request pay the full cold start, which is unacceptable for interactive traffic and fine for internal or batch use. Model loading also matters for rollbacks: rolling back to an old version is only fast if its weights are already cached.
SHADOW (mirror) CANARY (split)
client --> router --> model v1 --> response client --> router --95%--> model v1 --> response
| \--5%---> model v2 --> response
+--copy--> model v2 (result logged,
NEVER returned) watch: error rate, latency p99, business metric,
output distribution vs v1
compare v2 vs v1 offline; no user impact then 5% -> 25% -> 50% -> 100%, or roll backA model deploy is riskier than a code deploy because failures are often silent: the service returns 200s with plausible but worse predictions. Shadow serving sends a copy of live traffic to the candidate and discards or logs its output, so you can compare latency, errors and predictions against production without user impact. Its costs are roughly double compute for the mirrored share, and it cannot measure user-visible effects (clicks, conversions) because users never see the result; side effects (writes, emails, tool calls) must be disabled in the shadow path. Canary serving exposes a small percentage of real users to the new version and expands only if guard metrics hold, so it measures real behaviour but exposes those users to any regression. Typical practice is shadow first for correctness and latency, then canary for user impact, then a proper experiment for quality (see A/B Testing & Canary Deployments for Models and Model Monitoring & Drift Detection).
Queueing intuition (single queue, Poisson arrivals, exponential service, M/M/1):
utilisation rho = arrival rate / service rate
mean latency = service_time / (1 - rho)
1 - rho -> 0 makes latency explode: rho=0.5 -> 2x service, rho=0.9 -> 10x, rho=0.95 -> 20x
GPU sharing options (behaviour depends on GPU generation, driver and platform):
time-slicing several processes take turns on the GPU; simple, no memory isolation, latency jitter
MPS CUDA Multi-Process Service; kernels from several processes can run concurrently
MIG hardware partitions on supported GPUs (e.g. A100/H100) with dedicated memory + compute
one server, many models e.g. Triton loads several models and schedules across themQueueing explains why "we are only at 85% utilisation" is a warning, not reassurance: latency is not linear in load, and the tail grows even faster than the mean. The simulation below shows the p99 growing from 9x the service time at 50% utilisation to over 100x at 95%. Variability matters as much as load: LLM requests with widely varying output lengths behave like heavy-tailed service times, so they need lower target utilisation than fixed-cost requests. Pooling helps: several workers sharing one queue see much smaller tails than the same capacity split into separate queues, which is an argument for a shared queue or well-balanced routing rather than sticky per-replica queues (see Load Balancing).
GPU sharing raises utilisation of expensive hardware when models are small or traffic is low, but couples tenants: with time-slicing or MPS a noisy neighbour can inflate your latency, and memory is a hard limit shared by all resident models. MIG gives predictable isolation at the price of fixed partition sizes. Large LLMs generally take a whole GPU (or several, with tensor parallelism), so sharing is mostly for small and medium models and embedding or reranker models.
A seeded queue simulation: Poisson arrivals, exponential service with a 100 ms mean (an assumed service-time distribution, not a benchmark), FIFO, 200,000 requests per row. The first table sweeps utilisation for one worker and compares the simulated p99 with the exact M/M/1 result (latency is exponentially distributed with mean service/(1 - rho), so p99 = mean x ln 100). The second holds utilisation constant and changes the pool: one shared queue with 1, 4 or 8 workers, plus a fixed-service-time single worker.
import heapq, math, random
SERVICE_MS = 100.0 # mean service time per request (assumed)
def simulate(rho, servers, n=200_000, seed=1, deterministic=False):
"""FIFO queue, Poisson arrivals, `servers` identical workers. rho = utilisation per worker.
Returns sorted end-to-end latencies (queue wait + service) in ms."""
rng = random.Random(seed)
rate = rho * servers / SERVICE_MS # arrivals per ms
free = [0.0] * servers # min-heap of when each worker is next free
heapq.heapify(free)
t, lat = 0.0, []
for _ in range(n):
t += rng.expovariate(rate)
svc = SERVICE_MS if deterministic else rng.expovariate(1.0 / SERVICE_MS)
start = max(t, heapq.heappop(free)) # wait for the earliest-free worker
heapq.heappush(free, start + svc)
lat.append(start + svc - t)
lat.sort()
return lat
pct = lambda lat, p: lat[int(p * len(lat)) - 1]
print("1 worker, exponential service (mean 100 ms). M/M/1 theory: latency ~ Exp(mean = 100/(1-rho))")
print(" util mean_ms p50_ms p99_ms theory_p99 p99/service")
for rho in (0.30, 0.50, 0.70, 0.80, 0.90, 0.95):
lat = simulate(rho, 1)
th = SERVICE_MS / (1 - rho) * math.log(100) # exponential: p99 = mean * ln(100)
print(f" {rho:4.2f} {sum(lat)/len(lat):8.1f} {pct(lat,.50):8.1f} {pct(lat,.99):8.1f} {th:9.1f} {pct(lat,.99)/SERVICE_MS:8.1f}x")
print("\nSame utilisation, different pool: p99 latency (ms)")
print(" util 1 worker 4 workers 8 workers 1 worker, fixed 100 ms service")
for rho in (0.50, 0.80, 0.90, 0.95):
row = [pct(simulate(rho, s), .99) for s in (1, 4, 8)]
det = pct(simulate(rho, 1, deterministic=True), .99)
print(f" {rho:4.2f} {row[0]:8.1f} {row[1]:9.1f} {row[2]:9.1f} {det:14.1f}")
# Output:
# 1 worker, exponential service (mean 100 ms). M/M/1 theory: latency ~ Exp(mean = 100/(1-rho))
# util mean_ms p50_ms p99_ms theory_p99 p99/service
# 0.30 143.1 99.1 663.9 657.9 6.6x
# 0.50 200.0 138.7 918.0 921.0 9.2x
# 0.70 331.5 229.8 1516.9 1535.1 15.2x
# 0.80 497.4 341.3 2358.6 2302.6 23.6x
# 0.90 986.3 677.6 4857.2 4605.2 48.6x
# 0.95 2203.6 1357.0 11938.1 9210.3 119.4x
#
# Same utilisation, different pool: p99 latency (ms)
# util 1 worker 4 workers 8 workers 1 worker, fixed 100 ms service
# 0.50 918.0 474.9 460.1 432.6
# 0.80 2358.6 701.6 517.5 1111.9
# 0.90 4857.2 1270.3 724.6 2162.6
# 0.95 11938.1 3039.9 1583.6 4028.2Reading it: at 30% utilisation the p99 is already 6.6x the mean service time, because even a lightly loaded queue has a tail from variable service times and bursts of arrivals. From 50% to 90% utilisation p99 goes 918 ms to 4,857 ms, and at 95% the simulated value (11,938 ms) overshoots the 9,210 ms theory because near saturation the estimate needs far longer runs to converge; the qualitative message is what matters. In the second table, at 90% utilisation the p99 falls from 4,857 ms with one worker to 725 ms with a shared queue over 8 workers at the same per-worker utilisation, and a fixed 100 ms service time (no variability) more than halves the single-worker tail. So: keep utilisation modest, pool capacity behind one queue, and reduce service-time variance (cap output length, separate long and short requests).
Q: When would you choose batch inference over online inference?
A: When the result does not need to reflect data from the last few seconds, for example nightly scores, catalog embeddings or recommendation candidates. Batch runs at high, cheap utilisation on scheduled or spot capacity with no latency SLO, while online requires capacity sized for peak plus headroom. The trade-off is staleness and the risk of serving outdated scores; a hybrid (precompute offline, refine online) is common.
Q: Embedded model or separate model service?
A: Embed small, CPU-friendly models where the network hop and a second deployment would cost more than they give, and where the app team owns the model. Use a separate service when the model needs GPUs, is shared by several callers, has a different release cadence, or benefits from batching across callers. The cost of the service is a network dependency, more failure modes (timeouts, retries, circuit breaking) and an extra system to run.
Q: When is an async queue better than a synchronous API for inference?
A: When jobs are long (video, large documents), traffic is bursty, or callers do not need the answer immediately: the queue smooths bursts, permits retries and priority, and lets workers batch. The cost is job-state storage, a way to return results (polling or webhook), and at-least-once delivery, so workers must be idempotent. A queue does not add capacity: if arrivals exceed throughput, waiting time and queue size grow without bound, so you still need autoscaling and load shedding.
Q: What is a cold start in model serving and how do you reduce it?
A: It is the delay before a new replica can serve well: scheduling, pulling the container image, loading weights into memory (14 GB at 1-2 GB/s is roughly 7-14 seconds), initialising the runtime and warming kernels. Reduce it with cached images, weights on fast local storage, warm pools or minimum replicas, and a readiness probe that waits for warm-up. Because it takes tens of seconds to minutes, autoscaling must lead the load with queue-depth signals, not lag behind it.
Q: Explain shadow versus canary deployment for a model.
A: Shadow mirrors live requests to the new model and discards its output, so you compare latency, errors and predictions with zero user impact, but you double compute for the mirrored traffic and learn nothing about user behaviour. Canary sends a small real share of users to the new version and expands if guard metrics hold, which measures real impact but exposes those users to regressions. Shadow first, then canary, is a common sequence.
Q: Why does latency spike when utilisation goes from 80% to 95%?
A: In a simple queue, mean latency is service time divided by (1 - utilisation), so it goes from 5x to 20x service time, and the tail grows faster than the mean. In the M/M/1 simulation above, p99 rose from 2.4 s to about 12 s on a 100 ms service. Random bursts of arrivals create queues that take a long time to drain when there is little spare capacity, so target modest utilisation and scale out early.
Q: How would you share one GPU between several small models?
A: Options are running multiple models in one server such as Triton, CUDA time-slicing, MPS for concurrent kernels, or MIG hardware partitions on supported GPUs. Time-slicing and MPS raise utilisation but let one tenant hurt another's latency; MIG isolates but fixes partition sizes. Memory is the hard limit, so total resident model size plus activations must fit.
Follow-up: Your inference service has p50 of 80 ms but p99 of 2 seconds. Where do you look?
A: First queueing: check utilisation and queue wait over time, because tails come from waiting, not from service. Then variance sources: long inputs or outputs mixed with short ones, batch wait timers, GC pauses or cold replicas that were just added, noisy neighbours on a shared GPU, and retries amplifying load. Fixes include lowering target utilisation, separating request classes into pools, capping output length, warming replicas and hedged or timed-out requests with care.
Follow-up: How do you scale a GPU inference service when a new replica takes 3 minutes to become ready?
A: Scale earlier and on a leading signal: queue depth or requests in flight per replica, not CPU or latency after it has already degraded. Keep a minimum number of replicas or a warm pool sized to the burst you expect within one cold-start interval, and use admission control or load shedding to protect running replicas while capacity arrives. Ready-to-serve time, not steady-state throughput, sets how much headroom you need.
Follow-up: A shadow deployment shows the new model has better offline metrics on mirrored traffic. Is that enough to ship at 100%?
A: No. Shadow shows agreement, latency and errors, but users never saw the shadow output, so it cannot show the effect on clicks, conversions or complaints, and mirrored traffic misses feedback loops. Canary to a small share with guard metrics and a rollback path, and then an experiment if the business metric is what matters. Also check that latency under real load, not just on mirrored copies, still meets the SLO.
In a system design interview, state the serving mode and SLO first, then size the fleet with explicit assumptions. ILLUSTRATIVE example: peak 100 requests/s, a model service time of 100 ms on one worker, and a design rule of at most 60-70% utilisation to protect the tail; then workers needed is roughly 100 x 0.1 / 0.7, about 15 workers, plus one or two spares for failures and rolling updates, and about double that if you run a shadow copy at full mirroring. A recommendation system precomputes candidate embeddings in batch, serves ranking online from a model service, and streams click events to update features; an LLM gateway routes chat traffic synchronously with token streaming, sends long summarisation jobs to an async queue, and keeps warm replicas because cold starts take minutes. Mention the boundaries: timeouts and circuit breakers between app and model service, a separate GPU pool per latency class, model versions in a registry with canary rollout and quick rollback, and monitoring for latency percentiles, queue depth, GPU memory, error rate and model-quality signals. For capacity questions, connect to Capacity Planning & SLOs; for frameworks that implement these patterns (Triton, TorchServe, KServe, vLLM), see Model Serving Frameworks.
Every axis here trades simplicity and cost against latency, isolation or safety. Online serving buys freshness with idle capacity; batch serving buys cost with staleness. Model-as-a-service buys independent scaling and shared GPUs with a network hop and a new failure domain; embedding does the opposite. Async queues buy burst tolerance and retries with complexity and delayed answers; sync APIs buy simplicity and tie you to the model's latency. Warm pools and minimum replicas buy fast scale-out with idle spend; scale-to-zero does the reverse. Shadow and canary buy safety with duplicated compute and slower rollouts, and GPU sharing buys utilisation with noisy-neighbour risk. The wrong choices are usually the fashionable ones: a GPU model service for a tiny model that runs in-process, a sync API for 90-second jobs, running at 90% utilisation because it looks efficient, or scaling to zero an interactive endpoint. Start with the simplest architecture that meets the SLO, and add the next moving part only when a measured requirement demands it.