Multi-model serving means many different models share a pool of GPUs; multi-tenant serving means many customers (tenants) share the same models and hardware. They usually appear together in real platforms: an internal ML platform hosts dozens of team models, and an LLM API hosts thousands of customers, some with their own fine-tuned adapters. Interviewers probe it because the naive answer (one GPU per model, one deployment per customer) collapses on cost, while the naive sharing answer (one big queue) collapses on fairness and security. For a Backend or AI engineer the questions are concrete: what happens to a quiet customer when a loud one floods the queue, what does it cost to load a model that is not resident, and how do you guarantee one tenant never sees another's data.
Multi-model serving is running many different AI models on the same shared pool of machines, usually GPUs, instead of giving every model its own dedicated hardware. A company might have dozens of models, one for translation, one for image tagging, one for fraud scoring, and multi-model serving lets them all share the same pool, loading each model into memory only when it's actually needed rather than paying for idle hardware sitting behind a rarely-used model.
Multi-tenant serving is running one shared service for many different customers, or "tenants," at once, rather than standing up a separate copy of the system for every customer. The hard part is making sure one customer's heavy usage doesn't slow down or interfere with another customer's experience, and that the system stays fair and keeps each tenant's requests and data strictly separate even though everyone is sharing the same underlying models and machines.
Think of a shared commercial kitchen that rents space to several small food businesses. Multi-model serving is like one set of ovens and counters being used to cook many different businesses' recipes, brought out and put away as needed, instead of every business owning its own kitchen. Multi-tenant serving is the rule that keeps each business's ingredients, orders and payments separate, even though they are all cooking in the same room at the same time.
GPU memory (illustrative 80 GB card):
+-------------------------------------------------------------+
| model A weights | model B weights | KV cache + activations | <- every resident model
| 14 GB | 14 GB | (must stay large!) | also needs KV cache
+-------------------------------------------------------------+ headroom to serve traffic
Request for model C (not resident):
1. pick a victim (LRU) -> evict its weights + drop its KV/prefix cache
2. read weights (disk/object store -> host RAM -> GPU) <- seconds to tens of seconds
3. init engine, warm up kernels/graphs (framework dependent)
4. only now start prefill for the waiting requestWeights dominate memory: at 16-bit precision a model with N billion parameters needs roughly 2N GB (a 7B model is about 14 GB), so an 80 GB card cannot hold many large models and still leave room for KV cache, which is what actually determines how many concurrent requests each model can serve (see LLM Inference Fundamentals and GPU Hardware Utilization Basics). Quantization (Model Quantization & Compression) shrinks weights and lets more models fit. Loading is a bandwidth problem: assuming 14 GB read at an ILLUSTRATIVE 2 GB/s from local NVMe, that is about 7 s before any warm-up, and a network fetch from object storage can be slower. A request that triggers a cold load therefore waits seconds, not milliseconds, and it can stall other requests on the same GPU while memory is being reshuffled.
Multiplexing works when traffic is skewed: a few hot models stay resident and a long tail of rarely used models is loaded on demand and evicted LRU (least recently used). It fails in two ways. If the working set of models being used within a short window exceeds GPU memory, an LRU cache thrashes and nearly every request pays a cold load (the demo below shows a cyclic pattern over 4 models in 3 slots hitting 0%). And eviction is not free for the survivors either: dropped KV and prefix caches must be rebuilt. Systems commonly pin the hot models, route each model to a stable subset of replicas (so its cache is warm somewhere), and keep a small always-warm pool for latency-sensitive tenants. Frameworks such as Triton Inference Server and KServe's ModelMesh offer model load/unload management; check the current docs for the exact eviction and placement behaviour of the one you use, see Model Serving Architectures and Model Serving Frameworks.
One shared FIFO queue, one worker (capacity 100 req/s):
arrivals: noisy 250 req/s (burst) + quiet1 15 + quiet2 15 = 280 req/s offered
queue growth = 280 - 100 = 180 requests per second of burst
burst lasts 2 s -> ~360 requests backlog -> ~3.6 s to drain at 100 req/s
FIFO gives EVERY tenant that wait, because a quiet request joins the back of the same line.
Per-tenant queues + fair scheduler (round robin / weighted fair queueing):
noisy : [n n n n n n n n ...] \
quiet1 : [q] >--> scheduler picks the tenant with the least service so far
quiet2 : [q] /
quiet share = 100/3 = ~33 req/s each > 15 req/s offered -> quiet queues stay near emptyA noisy neighbour is not really a bug in the noisy tenant; it is a scheduling policy that treats requests, rather than tenants, as the unit of fairness. FIFO is work-conserving and simple but lets whoever sends the most requests own the queue. Per-tenant queues with a fair scheduler make the tenant the unit: each tenant with pending work gets at least its fair share of capacity, and idle capacity is still used by whoever has work (that is what "work-conserving" buys you over hard partitions). The demo below reproduces this with real numbers. Fairness does not create capacity: the noisy tenant's own wait gets worse under fair queueing (its p50 rose in the demo) because it is now the only one paying for its own overload, and total capacity is unchanged.
Real systems layer several controls, roughly in this order: admission control and rate limits at the edge, per-tenant token or concurrency quotas, a bounded per-tenant queue (reject with 429 and Retry-After instead of growing without limit; see Rate Limiting & Throttling), then fair scheduling inside the serving tier, then priority classes on top. LLM requests make "cost" uneven (a 4,000-token prompt is not a 20-token prompt), so fairness is usually measured in tokens or GPU time, not request count, and a scheduler that counts requests can still be gamed by a tenant sending few very expensive ones. Continuous batching engines schedule at the token-step level, so tenant fairness is often implemented above the engine by controlling which requests are admitted into it (see Batching & Dynamic Batching); whether an engine has native fairness features depends on the engine and version, so check its docs.
Priority classes (typical shape, names vary):
P0 interactive chat : small queue, tight deadline, may preempt/skip ahead
P1 API / paid tier : normal weight
P2 batch / offline eval : runs only in spare capacity, first to be shed or paused
Guard against starvation: aging (waiting time raises effective priority) or a guaranteed
minimum share for every class.
Isolation ladder (cost rises going down, blast radius shrinks):
shared pool, shared model -> quotas + fair queueing only
shared pool, per-tenant adapter -> + adapter-level access checks
dedicated replicas -> tenant gets its own GPUs, still same cluster
dedicated cluster/account -> regulated or very large customersStrict priority is dangerous: a busy P0 class starves P2 forever, so production schedulers add aging, reserved minimum shares, or weights (weighted fair queueing gives class or tenant X weight w and serves it in proportion to w). Weights encode business value, and they have a subtle cost the demo also shows: giving the noisy tenant weight 2 leaves each quiet tenant a smaller share (25 req/s each instead of 33), so quiet-tenant tail latency rises even though the quiet tenants are under their share, because Poisson arrivals at 15 req/s against a 25 req/s server still queue occasionally. Preemption of running work is expensive on GPUs (partial generations and KV state are lost or must be swapped), so it is mostly used at admission time, not mid-generation.
Hardware-level isolation options exist but come with limits. NVIDIA MIG (Multi-Instance GPU) can partition supported data-centre GPUs into isolated slices with dedicated memory and compute; time-slicing and MPS share a GPU with weaker or no memory isolation. Which technique is available depends on the GPU generation and the orchestration stack, and slicing reduces the size of model that fits, so treat it as an option to verify, not a default (see Kubernetes Basics for how GPUs are exposed to pods).
Leak paths to close in a shared serving stack:
- request/response logs and traces -> tenant-scoped, access controlled, retention per tenant
- prompt / prefix / KV caches -> scope cache keys by tenant (or by explicit opt-in);
shared prefix caches can leak via hit-timing (hedge: depends on engine)
- semantic / response caches -> key includes tenant id + model version
- fine-tuned weights and adapters -> authorization check at load AND at request time
- embeddings / vector store namespaces -> per-tenant index or mandatory tenant filter
LoRA adapter size (ILLUSTRATIVE, 7B-class model, hidden 4096, 32 layers,
adapters on q and v projections, rank r=16, 16-bit weights):
params per adapted matrix = 2 x 4096 x 16 = 131,072
x 2 matrices x 32 layers = 8,388,608 params
x 2 bytes = ~16.8 MB vs ~14 GB base modelPer-tenant fine-tunes are the reason LoRA matters for multi-tenancy. A LoRA adapter adds small low-rank matrices next to the frozen base weights, so hundreds of tenants can share one resident base model and each request selects its adapter, instead of holding hundreds of full 14 GB copies (compare Fine-tuning vs Prompting vs RAG). Some serving engines support multi-adapter serving (vLLM has LoRA support and research systems such as S-LoRA and LoRAX-style servers target thousands of adapters), but the maximum number of concurrent adapters, kernel efficiency when a batch mixes adapters, and merged-versus-unmerged latency vary by engine and version, so verify against current docs and measure before promising numbers. Adapters themselves can be LRU-cached in GPU memory just like models, at a much cheaper cold-load cost.
Routing ties these together. A router (often the gateway, see LLM Gateway & Multi-Provider Routing and Design an LLM API Gateway) authenticates the tenant (Authentication & Authorization), maps model name plus tenant to a pool, prefers replicas that already hold that model or adapter (locality-aware routing avoids cold loads), and enforces quotas. Metering rides on the same path: record tenant id, model, input and output tokens, and GPU-time or queue-time per request as an append-only usage event, keyed by request id so retries do not double bill (at-least-once events plus idempotent aggregation). Because continuous batching mixes tenants in one forward pass, GPU cost is usually attributed by token-weighted share rather than measured per request, which is an allocation policy, not a measurement (see Cost Optimization for AI Systems).
import random, heapq
from collections import deque
random.seed(42)
SERVICE_MS = 10 # one GPU worker, 10 ms per request -> capacity 100 req/s
SIM_MS = 6000
def poisson_arrivals(tenant, rate_fn, horizon):
"""Thinning-free approach: step 1 ms, Bernoulli arrival with p = rate/1000."""
out = []
for t in range(horizon):
if random.random() < rate_fn(t) / 1000.0:
out.append((t, tenant))
return out
def build_load():
noisy = lambda t: 250 if 2000 <= t < 4000 else 20 # burst: 250 req/s for 2 s
quiet = lambda t: 15
arr = (poisson_arrivals("noisy", noisy, SIM_MS)
+ poisson_arrivals("quiet1", quiet, SIM_MS)
+ poisson_arrivals("quiet2", quiet, SIM_MS))
arr.sort()
return arr
def simulate(arrivals, policy, weights=None):
weights = weights or {}
queues, vtime = {}, {}
fifo = deque()
waits = {}
i, now, n = 0, 0, len(arrivals)
while i < n or fifo or any(queues.values()):
while i < n and arrivals[i][0] <= now:
t, ten = arrivals[i]
if policy == "fifo":
fifo.append((t, ten))
else:
q = queues.setdefault(ten, deque())
vtime.setdefault(ten, 0.0)
if not q: # tenant becomes active: do not let idle time become a credit
active = [vtime[k] for k, qq in queues.items() if qq]
if active:
vtime[ten] = max(vtime[ten], min(active))
q.append(t)
i += 1
job = None
if policy == "fifo":
if fifo:
job = fifo.popleft()
else:
ready = [k for k, q in queues.items() if q]
if ready:
# pick tenant with smallest virtual time (weighted fair queueing, simplified)
ten = min(ready, key=lambda k: (vtime[k], k))
vtime[ten] += 1.0 / weights.get(ten, 1)
job = (queues[ten].popleft(), ten)
if job is None:
now = arrivals[i][0] if i < n else now + 1
continue
t, ten = job
waits.setdefault(ten, []).append(now - t) # queue wait (ms)
now += SERVICE_MS
return waits
def pct(xs, p):
xs = sorted(xs)
return xs[min(len(xs) - 1, int(p / 100 * len(xs)))]
arr = build_load()
counts = {}
for _, t in arr: counts[t] = counts.get(t, 0) + 1
print("requests:", counts, " capacity ~100 req/s")
for name, pol, w in [("FIFO", "fifo", None),
("Fair (equal weights)", "fair", None),
("Weighted (noisy=2, quiet=1)", "fair", {"noisy": 2})]:
w_ = simulate(arr, pol, w)
print(f"\n{name}")
for ten in ("quiet1", "quiet2", "noisy"):
xs = w_[ten]
print(f" {ten:7s} n={len(xs):4d} wait p50={pct(xs,50):5d} ms p99={pct(xs,99):5d} ms max={max(xs):5d} ms")
# Real output from python3 (seeded, deterministic):
# requests: {'noisy': 568, 'quiet1': 79, 'quiet2': 86} capacity ~100 req/s
#
# FIFO
# quiet1 n= 79 wait p50= 1688 ms p99= 3507 ms max= 3507 ms
# quiet2 n= 86 wait p50= 1470 ms p99= 3530 ms max= 3530 ms
# noisy n= 568 wait p50= 1818 ms p99= 3497 ms max= 3521 ms
#
# Fair (equal weights)
# quiet1 n= 79 wait p50= 11 ms p99= 86 ms max= 86 ms
# quiet2 n= 86 wait p50= 14 ms p99= 51 ms max= 51 ms
# noisy n= 568 wait p50= 2499 ms p99= 4010 ms max= 4041 ms
#
# Weighted (noisy=2, quiet=1)
# quiet1 n= 79 wait p50= 13 ms p99= 406 ms max= 406 ms
# quiet2 n= 86 wait p50= 15 ms p99= 312 ms max= 312 ms
# noisy n= 568 wait p50= 2499 ms p99= 4010 ms max= 4041 ms
import random
from collections import OrderedDict
random.seed(7)
MODELS = ["m-a", "m-b", "m-c", "m-d", "m-e", "m-f"]
WEIGHTS = [40, 25, 15, 10, 6, 4] # skewed traffic: a few hot models, a long tail
COLD_LOAD_S, WARM_S = 8.0, 0.05 # ILLUSTRATIVE numbers, not measurements
N = 5000
reqs = random.choices(MODELS, weights=WEIGHTS, k=N)
def run(slots):
cache, loads, total = OrderedDict(), 0, 0.0
for m in reqs:
if m in cache:
cache.move_to_end(m); total += WARM_S
else:
loads += 1; total += COLD_LOAD_S + WARM_S
if len(cache) >= slots:
cache.popitem(last=False) # evict least recently used
cache[m] = True
return loads, total / N
print("resident slots | cold loads | hit rate | mean latency (s)")
for slots in range(1, 7):
loads, mean = run(slots)
print(f"{slots:14d} | {loads:10d} | {1-loads/N:7.1%} | {mean:8.3f}")
# Thrashing: round-robin scan over 4 models with only 3 slots -> LRU never hits
cache, loads = OrderedDict(), 0
for i in range(400):
m = MODELS[i % 4]
if m in cache: cache.move_to_end(m)
else:
loads += 1
if len(cache) >= 3: cache.popitem(last=False)
cache[m] = True
print("cyclic access, 4 models / 3 slots: loads =", loads, "of 400 requests")
# Real output from python3 (seeded; latencies use ILLUSTRATIVE 8 s cold load / 0.05 s warm):
# resident slots | cold loads | hit rate | mean latency (s)
# 1 | 3651 | 27.0% | 5.892
# 2 | 2518 | 49.6% | 4.079
# 3 | 1555 | 68.9% | 2.538
# 4 | 839 | 83.2% | 1.392
# 5 | 337 | 93.3% | 0.589
# 6 | 6 | 99.9% | 0.060
# cyclic access, 4 models / 3 slots: loads = 400 of 400 requests
# Reading it: quiet tenants wait ~seconds under FIFO but ~tens of ms under fair queueing; the noisy
# tenant absorbs its own overload. Cold loads fall as resident slots rise, and cyclic access with
# too few slots defeats LRU entirely.Q: Why not just give every model or customer its own GPU?
A: Because utilization is usually low and skewed: a model that gets a few requests per minute still holds gigabytes of weights, so dedicated hardware is mostly idle and expensive. Sharing raises utilization but introduces cold loads, interference, and isolation risk. The right split is often shared pools for the long tail and dedicated replicas for high-traffic or high-compliance tenants.
Q: What is a noisy neighbour and how do you prevent it?
A: One tenant's load degrades others because they share a queue or GPU. Prevention is layered: per-tenant rate and token quotas, bounded queues that reject rather than grow, per-tenant queues with a fair or weighted scheduler, and priority classes so batch traffic yields to interactive traffic. In my simulation a 2 s burst at 250 req/s against a 100 req/s worker gave quiet tenants multi-second waits under FIFO and tens of milliseconds under fair queueing.
Q: If fair queueing protects quiet tenants, what does it cost?
A: It does not create capacity, so the noisy tenant now waits longer because it is the only one queuing behind its own backlog. It also needs per-tenant state and a fairness metric; counting requests is easy to game when request cost varies wildly, so LLM systems measure tokens or GPU time. And weights help high-value tenants but shrink everyone else's guaranteed share, which raises their tail latency.
Q: How do you decide which models stay loaded?
A: Start from traffic: pin the hot models (they carry most requests), let the tail be LRU-loaded, and size the resident set so the working set within a typical window fits. LRU is a reasonable default but thrashes on cyclic or scan-like patterns, so I would watch cold-load rate and p99 and consider frequency-aware eviction or a warm pool for latency-sensitive models.
Q: How would you serve per-tenant fine-tuned models cheaply?
A: Prefer parameter-efficient adapters (LoRA) over full fine-tunes: one shared frozen base model stays resident and each tenant's adapter is a small file (tens of MB in my illustrative estimate) loaded on demand and cached. The trade-offs are engine support and limits on concurrent adapters, some overhead when a batch mixes adapters, and quality limits versus a full fine-tune for some tasks. Adapter access must be authorized per tenant at load and request time.
Q: How do you guarantee one tenant cannot see another's data?
A: There is no single switch. Authenticate and authorize per request, scope every cache key, log store, vector index and adapter by tenant, avoid sharing prefix or semantic caches across tenants unless deliberately opted in, and consider dedicated replicas or MIG-style hardware partitions for high-risk tenants. Shared batching does not normally mix data between requests, but side channels (such as prefix-cache timing) are a known concern that depends on the engine, so I would state the residual risk instead of claiming it is zero.
Q: How do you bill fairly when many tenants share one batch?
A: Meter what is observable and attributable, which is tokens in and out per request, plus queue and generation time, and price on tokens with GPU cost amortized as an allocation policy. Emit usage events with request ids so retries do not double bill. Exact per-request GPU cost is not directly measurable under continuous batching.
Follow-up: A tenant complains of random slow requests only right after idle periods. Diagnose it.
A: That pattern suggests cold loads: the model or adapter was evicted during idle time and the first request pays the load cost. Confirm by correlating slow requests with load events and the time since last request for that tenant or model. Fixes are a minimum-warm replica or pinning for that tenant, keep-alive traffic (at a cost), or locality-aware routing; the trade-off is paying for idle GPU memory.
Follow-up: Strict priority for paid tenants is starving free-tier users completely. What do you change?
A: Move from strict priority to weighted fair queueing or guarantee a floor for each class, and add aging so waiting time raises effective priority. You keep paid tenants faster while bounding worst-case delay for free tier. The cost is that paid tenants lose some headroom under saturation, so the floor must be sized deliberately.
Follow-up: Your LRU model cache thrashes at peak. What are your options?
A: First confirm the active set really exceeds memory; then add capacity, quantize models to fit more, shard the model set across replicas so each replica sees a smaller working set (consistent-hash routing by model), pin the hottest models, or fall back to a smaller shared model for tail traffic. Each has a cost: more GPUs, some quality loss, or routing complexity and less even load.
In an "LLM platform for many customers" or "internal ML inference platform" design, name the layers: gateway with authentication and per-tenant quotas, a router with model and adapter locality, per-tenant bounded queues feeding a fair scheduler in front of the engines, a model registry and loader, and a metering pipeline. State assumptions and do the arithmetic aloud: for example, ASSUME 200 tenants, 20 base models, 80 GB GPUs and 14 GB per model; the hot four models are pinned, the tail is LRU, and a cold load costs roughly 7 s if weights stream at 2 GB/s (ILLUSTRATIVE), so tail traffic has a documented latency floor and hot models do not. Size the fair-share floor from offered load: if a worker serves about 100 req/s and 50 tenants are active, an equal share is 2 req/s, which is fine for most tenants and tells you exactly which large tenants need weights or dedicated replicas. Mention that data isolation, quotas and observability per tenant (Observability for AI Systems) are part of the design rather than afterthoughts, and that autoscaling (Autoscaling for Inference Workloads) has to scale per model pool, not just per cluster.
Sharing GPUs across models and tenants trades utilization and cost for latency variance, complexity and risk: cold loads add seconds to the first request after eviction, fair scheduling protects quiet tenants but worsens the noisy tenant's wait, weights and priorities encode business decisions that shrink someone's share, and every shared cache or batch is a possible isolation question. Dedicated replicas are simpler to reason about and give predictable latency and a clean blast radius, but they waste memory on idle models and multiply operational cost. Multi-tenancy is the wrong choice when a tenant has strict compliance or latency requirements that shared infrastructure cannot credibly meet, when a handful of models take almost all traffic (just replicate them, no multiplexing needed), or when the team cannot yet operate the quota, metering and isolation machinery it requires; in those cases, start with dedicated pools and share only the long tail.