A model file is not a service. Between "a trained network" and "an endpoint that answers 500 requests per second within 100 ms" sit model loading, request queueing, batching, hardware placement, version management, health checks, metrics and a wire protocol. A serving framework packages those concerns so that you do not rewrite them around a Flask handler that calls model.predict() once per request, which is the classic way to leave most of a GPU idle.
Interviewers ask about serving frameworks to see whether you know what problem each layer solves and can choose without cargo-culting a tool name. The good answer names the responsibilities (batching, concurrency, model lifecycle, observability), maps a few frameworks to their sweet spots with humility, and reasons from constraints: model type, hardware, latency SLO and how much operational complexity the team can carry. This page is the tooling companion to model serving architectures (the topology: online, batch, streaming) and batching and dynamic batching (the mechanism most frameworks implement).
A model serving framework is the software that turns a trained AI model into a live, reliable web service. On its own, a trained model is just a file full of numbers, it can't handle two requests at once, doesn't know how to group similar requests together for efficiency, and has no way to report whether it's healthy. A serving framework (like NVIDIA Triton, TorchServe, TensorFlow Serving or BentoML) wraps that file with everything needed to answer real traffic: a network endpoint, request queueing, automatic batching, hardware scheduling, version management and health checks, so an engineering team doesn't have to build all of that from scratch for every model.
Think of a busy restaurant kitchen. The chef (the trained model) knows how to cook, but a working kitchen also needs someone taking orders, a system for grouping similar orders so the grill isn't cooking one burger at a time, an expediter checking each plate is ready before it goes out, and a manager watching whether the kitchen is falling behind. A serving framework is that entire kitchen operation built around the chef, so the chef can just focus on cooking.
client --HTTP/gRPC--> [ protocol front-end ] --> [ per-model request queue ] --> [ scheduler / batcher ]
|
model repository (versions) --> [ model loader / cache ] --> [ execution instances (CPU/GPU) ]
|
[ metrics: queue time, compute time, batch size, errors ] <--- exported (e.g. Prometheus format)
Responsibility Why it exists
------------------------ -----------------------------------------------------------------
Model loading / lifecycle load once, warm up, hot-swap versions without dropping traffic
Dynamic batching GPU cost is mostly per launch; grouping requests amortises it
Concurrency / instances run N copies of a model, or several models, on one accelerator
Multi-model hosting many models behind one endpoint, load/unload by demand
Protocol standardisation one API (REST/gRPC, often a v2-style inference protocol) for all models
Metrics and health readiness, latency histograms, queue depth for autoscaling and SLOs
Pre/post-processing tokenisation, resizing, decoding; sometimes as pipelines/ensembles
Little's law for sizing (illustrative): concurrent in-flight = arrival rate x latency
200 req/s x 0.050 s = 10 requests in flight -> the batcher can typically collect a few of them.The core insight is that inference hardware rewards batching: a GPU kernel launch and weight read has a large fixed cost, so processing 8 inputs together costs far less than 8 times processing one (the demo below models exactly that). A per-request Python handler forfeits that. Frameworks add a queue in front of the model and a scheduler that groups queued requests until either a maximum batch size or a maximum wait time is reached, which is the latency-for-throughput trade covered in batching and dynamic batching. The wait budget must come out of your latency SLO, not on top of it.
Where it breaks: the framework's queue hides overload until latency has already blown up, so you need queue-depth limits and load shedding at the edge; a model that loads slowly (tens of gigabytes of weights) makes scale-out and rollouts slow, so cold starts have to be planned; and a shared GPU across models can let one noisy model starve another unless you set instance groups and priorities (see multi-model and multi-tenant serving). Metrics should separate queue time from compute time, otherwise you cannot tell "model is slow" from "batcher is waiting".
Framework Sweet spot (generally) Watch out for
--------------- ------------------------------------------------- --------------------------------------
Triton (NVIDIA) multi-framework, GPU-heavy, dynamic batching, config surface area; most benefit on
model ensembles, concurrent model execution NVIDIA hardware; steeper ops learning
TensorFlow TF SavedModel serving, versioned model directories, TF-centric; less natural for mixed
Serving gRPC/REST, request batching fleets
TorchServe PyTorch model archives, handlers, simple REST check current maintenance status of
the project before adopting
vLLM LLM serving: continuous batching, paged KV cache, LLM-only; GPU memory tuning matters
OpenAI-compatible HTTP API
TGI (Hugging LLM serving in the Hugging Face ecosystem, overlaps vLLM; check current status
Face) token streaming and supported models
TensorRT-LLM NVIDIA-optimised LLM inference, often paired with NVIDIA-specific; build/compile steps
Triton tied to GPU and version
BentoML package model + Python pre/post code as a "bento", Python-level service; you still own
adaptive batching, developer experience the infrastructure choices
Ray Serve Python-first, model composition/pipelines, need a Ray cluster; more moving
replica autoscaling on Ray parts than one binary
KServe Kubernetes-native InferenceService CRD wrapping needs Kubernetes competence;
runtimes (Triton, TF, etc.), canary, autoscaling itself a layer over other servers
Rule of thumb: runtimes (Triton, vLLM, TF Serving) EXECUTE models;
platforms (KServe, Ray Serve, BentoML cloud) ORCHESTRATE many such servers.Do not treat that table as a ranking; it is a map, and every cell should be checked against current documentation because these projects evolve, merge features and change maintenance status. The useful distinction is between inference runtimes that own the batching and execution of a model on a device, and serving platforms that own deployment, scaling and routing. KServe on Kubernetes commonly delegates the actual inference to Triton or another runtime; Ray Serve and BentoML sit closer to the application, letting you compose several models and business logic in Python. See Kubernetes basics for the substrate and autoscaling for inference workloads for how these platforms scale replicas.
No numbers are quoted for "which is fastest" because throughput depends on the model, batch size, precision, hardware and configuration; published comparisons usually measure one setup. The defensible interview position is: benchmark your own model, at your batch and latency targets, on your target hardware, and compare p99 latency and cost per thousand requests, not headline peak throughput.
Train in PyTorch/TF --export--> ONNX (portable graph) --> ONNX Runtime (CPU/GPU/other execution providers)
\--> TensorRT engine (NVIDIA GPU, compiled for a specific target)
portability speed potential flexibility after build risk
native PyTorch high baseline highest slower, heavier runtime
ONNX Runtime high often better good unsupported ops, dynamic-shape quirks
TensorRT engine low often best lowest (rebuild to change) tied to GPU arch + TensorRT version
TensorRT builds an optimised engine (layer fusion, kernel selection, optional FP16/INT8 precision).
- engine is generally specific to the GPU architecture and TensorRT version it was built on
- INT8 needs calibration data and an accuracy check; FP16 usually a smaller risk but still verify
- dynamic input shapes need declared min/opt/max profiles
Conversion checklist:
1. export with representative inputs and dynamic axes declared
2. compare outputs vs the original model on a golden set within a tolerance
3. re-run the accuracy/eval suite, not just a few samples
4. record the conversion toolchain versions in the model manifestConversion is an optimisation you pay for in risk. Exporting to ONNX decouples the model from the training framework and lets ONNX Runtime pick execution providers for the hardware, but not every operator or control-flow construct exports cleanly, and custom ops need extra work. Compiling to TensorRT can bring meaningful speedups by fusing layers and lowering precision, at the price of a build step, a hardware- and version-specific artifact, and sometimes small numerical differences. How large the gain is depends on the model and precision, so measure instead of assuming.
The operational implication is that the converted artifact is a build product: it belongs in the registry with a manifest recording the source model version, converter versions, precision and target GPU, and it must pass the same evaluation gate as the original. Quantisation choices are covered in model quantisation and compression. A frequent production incident is a converted engine built on one GPU type being deployed to another, or a driver upgrade invalidating engines, so rebuild engines in CI per target rather than shipping them from a laptop.
Classic model (ResNet, BERT classifier, XGBoost): one forward pass per request, fixed-ish cost
-> batch requests, run once, return. Dynamic batching is the main lever.
LLM: two phases, variable length, autoregressive
prefill : process the whole prompt in parallel (compute-bound)
decode : generate 1 token per step, many steps (generally memory-bandwidth bound)
Static batching: wait for a batch, run until the LONGEST sequence finishes -> finished slots sit idle
Continuous batching: after each decode step, evict finished sequences and admit new ones -> slots stay busy
(implemented by LLM-specific servers such as vLLM, TGI, TensorRT-LLM)
KV cache per token = 2 (K and V) x layers x kv_heads x head_dim x bytes
Illustrative 7B-class config: 32 layers, 32 KV heads, head_dim 128, fp16 (2 bytes)
2 x 32 x 32 x 128 x 2 = 524,288 bytes = 0.5 MiB per token
4,096-token sequence = ~2 GiB of KV cache for ONE request (models using grouped-query attention need less)
-> memory, not compute, often limits how many concurrent sequences fit.
Paged KV cache (vLLM's PagedAttention) allocates the cache in blocks to cut fragmentation and waste.A classic model server can treat each request as a fixed-size unit of work; an LLM server cannot. Output length is unknown in advance, requests finish at different times, and every active sequence holds a growing KV cache in GPU memory. That is why general-purpose batchers, which wait for a batch to complete, waste capacity for LLMs, and why dedicated engines schedule at the granularity of a single decode step (continuous batching) and manage KV memory in blocks. Streaming tokens to the client (see streaming responses) is also part of the contract: time to first token is dominated by queueing plus prefill, while inter-token latency reflects decode speed and how many sequences share the step.
The practical result is two different metric sets. For classic models you watch p99 latency, batch-size distribution and GPU utilisation. For LLMs you add time to first token, inter-token latency, tokens per second, queue depth, and KV-cache utilisation, and you cap concurrent sequences or context length to avoid running out of KV memory. Many platforms let you put an LLM engine behind the same orchestration layer (for example KServe or Ray Serve) so operations stay uniform even though the engine differs; see LLM inference fundamentals for the phases in more depth and GPU utilisation basics for why decode is bandwidth-limited.
Decision inputs Typical implication (not a rule)
--------------------------------------- ----------------------------------------------------------
LLM / autoregressive? LLM engine (vLLM, TGI, TensorRT-LLM) before a generic server
Mixed frameworks, GPU-heavy, many models Triton (multi-framework, dynamic batching, ensembles)
Pure PyTorch/TF, small team, modest load Framework-native server or BentoML; keep it simple
Python business logic + several models BentoML or Ray Serve (composition in Python)
Already on Kubernetes at scale KServe (or plain Deployments) wrapping one of the runtimes
CPU-only, tiny models (trees, linear) a plain service or ONNX Runtime; a GPU server is overkill
Tight latency SLO (say p99 under 50 ms) small batch window (a few ms), pre-warmed instances, load shedding
Throughput/cost focused, latency relaxed larger batches, longer wait, quantisation
Batch-window sanity check (illustrative): SLO p99 = 100 ms, compute for batch = 40 ms, network+pre/post = 20 ms
-> queue wait budget = 100 - 40 - 20 = 40 ms -> max_wait of ~10-20 ms leaves headroom for variance.Choose by constraints, not by fame. Model type comes first (LLMs need continuous batching and KV-cache management); hardware second (Triton and TensorRT shine on NVIDIA GPUs, ONNX Runtime and plain services cover CPU and mixed fleets); then the latency SLO, which fixes the batching window; and finally the team, because the best runtime nobody can operate is worse than a slightly slower one they understand. Total cost of ownership includes on-call load: KServe plus Knative plus Triton is powerful and also several systems to upgrade.
A staged path avoids over-engineering: begin with a simple framework-native or BentoML service, measure real traffic, and adopt Triton or an LLM engine when profiling shows the GPU idle, per-request overhead dominating, or multi-model packing needed. Keep the model behind a stable API so the runtime can change later without touching clients, and record the serving stack version in the model manifest for reproducibility.
predict() supplies none of it.A tiny model server built from the pieces above: a registry that loads models once, per-model request queues, micro-batching by size or wait time, and metric counters. Inference is simulated with virtual milliseconds (launch overhead plus a per-item cost), and compute is not serialised across batches, so the numbers show the amortisation effect, not a real GPU. Arrivals are deterministic, so the output is reproducible.
from collections import deque, defaultdict
class Model:
"""Simulated model: cost = fixed launch overhead + per-item cost (ms of virtual time)."""
def __init__(self, name, overhead_ms, per_item_ms, max_batch):
self.name, self.overhead, self.per_item, self.max_batch = name, overhead_ms, per_item_ms, max_batch
def infer_batch(self, inputs):
latency = self.overhead + self.per_item * len(inputs)
return [f"{self.name}({x})" for x in inputs], latency
class Server:
def __init__(self, max_wait_ms):
self.models, self.queues = {}, {}
self.max_wait = max_wait_ms
self.m = defaultdict(int)
self.latencies = []
def load(self, model): # registry: model loaded once, reused per request
self.models[model.name] = model; self.queues[model.name] = deque(); self.m["models_loaded"] += 1
def submit(self, name, x, t):
if name not in self.models: self.m["errors_unknown_model"] += 1; return False
self.queues[name].append((x, t)); self.m["requests_total"] += 1; return True
def step(self, now):
"""Micro-batching: flush a queue when it is full OR its oldest request waited max_wait."""
for name, q in self.queues.items():
mdl = self.models[name]
while q and (len(q) >= mdl.max_batch or now - q[0][1] >= self.max_wait):
batch = [q.popleft() for _ in range(min(mdl.max_batch, len(q)))]
_, lat = mdl.infer_batch([x for x, _ in batch])
self.m["batches_total"] += 1; self.m["compute_ms_total"] += lat; self.m["items_total"] += len(batch)
for _, t in batch: self.latencies.append(now - t + lat) # queue wait + compute
def report(self):
l = sorted(self.latencies)
return {**self.m, "avg_batch": round(self.m["items_total"]/self.m["batches_total"], 2),
"p50_ms": l[len(l)//2], "max_ms": l[-1]}
def run(max_wait, max_batch):
s = Server(max_wait)
s.load(Model("resnet", overhead_ms=20, per_item_ms=2, max_batch=max_batch))
s.load(Model("bert", overhead_ms=30, per_item_ms=5, max_batch=max_batch))
# deterministic arrivals: one request every 3 ms alternating models, 60 requests
for i in range(60):
t = i * 3
s.submit("resnet" if i % 2 == 0 else "bert", i, t)
s.step(t)
s.step(10_000) # drain
s.submit("gpt", 0, 0) # unknown model -> counted as error
return s.report()
for w, b in [(0, 1), (10, 8), (30, 16)]:
print(f"max_wait={w:>2}ms max_batch={b:>2} ->", run(w, b))
# Output:
# max_wait= 0ms max_batch= 1 -> {'models_loaded': 2, 'requests_total': 60, 'batches_total': 60, 'compute_ms_total': 1710, 'items_total': 60, 'errors_unknown_model': 1, 'avg_batch': 1.0, 'p50_ms': 35, 'max_ms': 35}
# max_wait=10ms max_batch= 8 -> {'models_loaded': 2, 'requests_total': 60, 'batches_total': 20, 'compute_ms_total': 710, 'items_total': 60, 'errors_unknown_model': 1, 'avg_batch': 3.0, 'p50_ms': 45, 'max_ms': 57}
# max_wait=30ms max_batch=16 -> {'models_loaded': 2, 'requests_total': 60, 'batches_total': 10, 'compute_ms_total': 460, 'items_total': 60, 'errors_unknown_model': 1, 'avg_batch': 6.0, 'p50_ms': 62, 'max_ms': 90}Read the trade-off off the three lines: going from no batching to batches averaging 6 cuts total simulated compute from 1710 ms to 460 ms for the same 60 requests (higher throughput per unit of hardware), while median latency rises from 35 ms to 62 ms because requests now wait in the queue. That is the same lever real frameworks expose (Triton's dynamic batching delay, BentoML's adaptive batching, and similar) under different names. A Triton model configuration looks roughly like the following; it is a config sketch with no measured output, and field names should be checked against the current documentation:
# config.pbtxt (illustrative, not executed here)
name: "resnet"
platform: "onnxruntime_onnx"
max_batch_size: 16
dynamic_batching {
preferred_batch_size: [ 4, 8 ]
max_queue_delay_microseconds: 5000 # wait up to 5 ms to form a batch
}
instance_group [ { count: 2 kind: KIND_GPU } ]Q: Why not just wrap the model in Flask or FastAPI?
A: For prototypes and low traffic it is fine, but a per-request handler runs one input at a time, loads or guards the model by hand, and gives you no batching, no version hot-swap, no standard metrics and no multi-model scheduling. On a GPU that usually means low utilisation and a higher cost per request. Once traffic or model count grows, a serving framework buys those features; the trade-off is another component to configure and operate.
Q: How does dynamic batching work and what does it cost?
A: The server holds requests in a queue and dispatches a batch when it reaches a maximum size or the oldest request has waited a maximum delay. Because kernel launch and weight loads have a large fixed cost, a batch of 8 is far cheaper than eight singles, raising throughput. The cost is added queueing latency up to the max delay, so that delay must fit inside the SLO budget after subtracting compute and network time.
Q: Triton versus TorchServe versus BentoML: how would you choose?
A: Triton when I have GPU-heavy, possibly mixed-framework models and want dynamic batching, concurrent instances and ensembles, accepting a larger configuration surface and NVIDIA affinity. A framework-native server or BentoML when the team is one ecosystem, the load is moderate and developer experience matters, with BentoML easier for bundling Python pre/post-processing. I would also check each project's current maintenance status and benchmark my own model before committing.
Q: What is different about serving an LLM?
A: Output length is variable and generation is autoregressive, so requests finish at different times and each holds a growing KV cache. Static batching wastes slots waiting for the longest sequence, so LLM engines such as vLLM use continuous batching and manage the cache in blocks. You also track time to first token, inter-token latency and cache utilisation, and you cap concurrency or context to stay within GPU memory.
Q: When is ONNX or TensorRT worth it?
A: When profiling shows the model is a meaningful cost driver and you have NVIDIA GPUs (for TensorRT) or need portability across hardware (for ONNX Runtime). The gain depends on model and precision, so measure. The costs are conversion failures on unsupported ops, small numerical differences, a build pipeline, and engine artifacts tied to a GPU architecture and TensorRT version that must be rebuilt in CI.
Q: How do you roll out a new model version safely on these servers?
A: Put the new version in the model repository, let the server load and warm it alongside the old one, then shift a small share of traffic through the platform layer (KServe canary, an API gateway or weighted routing) and compare latency, errors and business metrics before increasing it. Keep the old version loaded for instant rollback. Frameworks that hot-swap versions still need warm-up to avoid a cold-start latency spike.
Q: Which metrics would you export from a serving layer?
A: Request rate, error rate by class, latency split into queue time and compute time, batch-size distribution, model load state, GPU utilisation and memory, and for LLMs time to first token, inter-token latency, tokens per second and KV-cache usage. Queue depth is the best autoscaling and load-shedding signal because it rises before latency does. See observability for AI systems.
Follow-up: GPU utilisation shows 30% even though latency is at the SLO. What now?
A: The SLO may be capping the batch window, so utilisation is low by design; check batch-size distribution and queue time. Options are running multiple model instances or co-locating models on the GPU, raising the batch window if the SLO has slack, using a smaller or cheaper GPU, or quantising. Do not chase utilisation blindly: 30% with headroom for bursts may be the right cost for a strict latency target.
Follow-up: A converted TensorRT engine gives slightly different predictions from the PyTorch model. Acceptable?
A: Small differences are expected from fused kernels and reduced precision, so the question is whether the differences stay within a tolerance on a golden set and whether the evaluation metrics on the real eval suite hold. If a metric regresses, fall back to higher precision for sensitive layers or to FP16 or FP32. Log the conversion details in the manifest and gate promotion on the eval result.
Follow-up: How would you host 200 small models on a few GPUs?
A: Use a multi-model server that loads and unloads models on demand from a model repository, share GPUs by instance groups, and route by model id, accepting cold-start latency for rarely used models and using an LRU cache of loaded models. Group hot models together and keep tail models on CPU if they are small. The failure mode is thrash when the working set exceeds GPU memory, so measure hit rate; details in multi-model and multi-tenant serving.
In a design like a real-time recommendation or an LLM API gateway, the serving framework is the box behind the load balancer that turns "N models on M GPUs" into an SLO-bound service. Estimate with stated assumptions: at 2,000 requests per second peak with a 50 ms p99 target, Little's law gives roughly 2,000 x 0.05 = 100 requests in flight at any moment, which across replicas and a batch window of a few milliseconds yields batches of a handful per replica; size replicas from a load test of your model, then add headroom for failure and bursts (commonly 30 to 50 percent, an assumption to state). For LLMs, capacity is governed by KV memory and tokens per second: the 0.5 MiB per token illustration above means 100 concurrent 4,096-token sequences need about 200 GiB of KV cache, more than one GPU, which is why context caps, grouped-query attention, quantised caches and multiple GPUs enter the design. The framework choice also shapes the surrounding system: a Kubernetes shop typically wraps a runtime in KServe or Deployments with queue-depth autoscaling (autoscaling for inference), puts an API gateway in front for auth and rate limits, and uses canary routing tied to the registry. Version, conversion and image metadata flow from the model registry into the deployment so the running stack is always attributable.
Adopting a serving framework trades simplicity for capability: you gain batching, hot version swap, multi-model packing and consistent metrics, and you pay in configuration surface, another layer to upgrade and debug, and possible lock-in to one vendor's format or hardware (Triton and TensorRT lean NVIDIA; framework-native servers lean one ecosystem). Framework-agnostic runtimes maximise flexibility and GPU efficiency; developer-centric tools such as BentoML and Ray Serve maximise iteration speed and composition but leave more infrastructure decisions to you; platforms like KServe standardise operations across teams but demand Kubernetes maturity. Conversion to ONNX or TensorRT adds speed and shrinks cost when a model is a real cost driver, and adds risk through unsupported ops, numerical drift and hardware-tied artifacts. The wrong choice is reaching for the heaviest stack for a low-traffic CPU model, or the opposite, serving a GPU model through a naive per-request handler and paying for idle silicon. Start with the simplest thing that meets the SLO, measure, and escalate only when the measurements justify it.