An LLM gateway is a service that sits between your applications and one or more model providers (hosted APIs or your own inference clusters) and turns "call a model" into a managed, observable, controllable operation. Interviewers probe it because it is where reliability, cost and security of an AI product actually get decided: providers have outages and rate limits, prices and latencies differ by an order of magnitude between models, and nobody wants every team pasting API keys into its own service. For a Backend or AI engineer the questions are concrete: what belongs in the gateway, how do you route and fail over without making things worse, and how do you avoid the gateway becoming the single point of failure it was built to remove. The end-to-end design version is in Design an LLM API Gateway; this page is the routing and multi-provider mechanics.
An LLM gateway is a middleman service that every part of a company's software talks to instead of calling an AI provider (like OpenAI or Anthropic) directly. It's the single front desk that checks who's asking, keeps track of spending, and forwards the actual request onward, so individual apps and teams don't each need to handle logins, billing, and outages on their own.
Multi-provider routing is the logic that decides, for each request, which AI company's model should actually answer it & a company might use a cheap, fast model for simple questions and a more expensive, powerful one for hard questions, or switch providers automatically if one is having an outage. It's the same idea as a delivery app that picks whichever driver, restaurant, or route will get your food to you fastest and cheapest at that particular moment.
Think of a hotel concierge desk. Guests don't call the restaurant, the car service, and the dry cleaner directly & they tell the concierge what they need, and the concierge decides which partner business to call, keeps a running bill for the guest's room, and if their usual dry cleaner is closed, quietly sends the order to a backup one instead. The guest never notices the switch. An LLM gateway plays that exact concierge role between a company's software and the various AI companies it works with.
apps / agents / batch jobs
| one internal API (one schema, virtual API key per team)
v
+---------------------------------------------------------------+
| LLM GATEWAY |
| 1 authn/authz who is this? which models may they use? |
| 2 quotas + limits per-tenant RPM/TPM/budget (429 early) |
| 3 cache lookup exact / semantic, tenant-scoped |
| 4 router pick provider + model (policy) |
| 5 adapter translate schema, tokenizer, params |
| 6 resilience timeout, retry, fallback, breaker |
| 7 stream relay forward events without buffering |
| 8 accounting tokens in/out, cost, latency, provider |
+---------------------------------------------------------------+
| | |
provider A provider B self-hosted (vLLM etc.)The gateway earns its place by centralising concerns that are expensive to get right ten times. Authentication maps a caller to a tenant and to allowed models; quotas and budgets stop one runaway agent loop from consuming the whole provider limit (the mechanics are in Rate Limiting and Throttling); the adapter hides provider differences; resilience logic lives once instead of in every client; and every call emits the same log and metric shape, which is what makes cost attribution ("which team, which feature, which prompt version spent this") possible (see Observability for AI Systems and Cost Optimization for AI Systems). Its cost is a new tier on the critical path of every AI feature: an extra network hop, an extra deploy target, and a new outage domain, so it has to be stateless where possible, horizontally scaled, and boringly reliable.
Policy Rule Watch out for
---------------- ------------------------------------- ------------------------------------
static route "summarize" to model X no adaptation to outages
cost-first cheapest model that meets the tier cheap model quality regression
latency-first lowest recent p95 / TTFT herd effect: everyone piles onto
the "fastest" provider, it slows down
quality-tier hard prompts -> strong model, classifier errors; needs evals
easy prompts -> small model
weighted / canary send 5% to the new model or provider need per-variant metrics and a kill switch
fallback chain try A, on failure/timeout try B, C latency stacks up; B/C must be
pre-validated for the same task
Typical composition (order matters):
1. filter : models the tenant may use, that support the needed features
(tools, JSON mode, context length for THIS prompt)
2. filter : providers whose circuit is not open, with quota headroom
3. choose : policy (cost / latency / quality / weights) among what is left
4. chain : the remaining candidates, in order, become the fallback listThe important idea is that "which model" is really two decisions: a capability decision (can this model do the task and fit the context) and an optimisation decision (among those that can, which is cheapest or fastest right now). Doing them in that order keeps you from failing over to a model that silently cannot handle tool calls or the prompt length. Latency-based routing needs damping (moving averages, minimum traffic to every target) or it oscillates. Quality-based routing is only as good as your offline evaluation, so it pairs with the rollout discipline in A/B Testing and Canary Deployments for Models: send a small weighted share to a candidate, compare quality and error metrics per variant, and keep an instant kill switch. Fallbacks are not free either: a fallback model can answer differently in tone, format and refusals, which downstream parsers may not tolerate, so treat the fallback list as tested configuration, not a guess.
Breaker per provider (per model/region if they fail independently):
CLOSED --(N consecutive failures)--> OPEN --(cooldown elapsed)--> HALF-OPEN
^ | |
| +-- calls skipped instantly +-- allow ONE probe
+-------- probe succeeds -------------------------------------------+ probe fails -> OPEN
Why not just retry? During an outage every call to the dead provider hangs until the timeout.
attempts on dead provider x timeout (say 2 s) x retries = seconds of latency added
to EVERY request, plus retry traffic hitting a provider that is already struggling.
Retry only what is safe and useful:
retry : timeout, connection reset, 429 (honour Retry-After), 500/502/503/529-style overload
do not : 400/401/403/422 (your bug), content-filter refusals, anything after tokens were streamed
always : exponential backoff + jitter, a retry BUDGET, and an overall request deadlineThe Try It Yourself simulation compares four configurations over the same virtual traffic in which the cheapest provider is hard-down for a 30-second window. With the cheapest provider only and no retries, about a third of requests fail; retries alone barely help during a hard outage and push p95 up because each retry waits for another timeout; a fallback chain reaches 100% success but every request in the outage window still pays the dead provider's timeout first; adding a circuit breaker removes almost all calls into the outage and brings p95 back close to normal while cost rises only by the difference in per-request price. The lesson is the interaction: retries fix transient errors, fallback fixes sustained failure, and the breaker is what stops fallback from being slow. Retry storms are the failure to guard against, since three attempts per layer across three layers is 27 calls for one user request; retry at one layer (usually the gateway), bound it with a budget (for example retries capped at a fraction of traffic), and see Fault Tolerance and Circuit Breakers and Timeouts, Retries and Backoff.
Difference Why it bites Gateway approach
----------------------- ----------------------------------------------- ------------------------------------
request/response schema message roles, system prompt placement, tool canonical internal schema + one
call and tool result formats differ adapter per provider
streaming events different event names and end-of-stream signals normalise to one event grammar
tokenizers same text = different token counts per provider count with provider's own numbers;
-> quotas, context checks, cost all differ trust returned `usage` for billing
context / output limits different maximums per model check prompt length BEFORE choosing
rate limits RPM, TPM, concurrency; differ by tier/model track headroom per provider key
features JSON mode, tools, vision, prompt caching capability matrix; route by need
errors status codes and retry hints are inconsistent map to a small internal error set
parameters temperature ranges, stop sequences, seed clamp/translate; document gapsNormalisation has an honest limit: a canonical schema is the lowest common denominator plus escape hatches. If it exposes only what every provider supports, you lose useful features; if it passes provider-specific parameters through, callers become coupled again. A workable compromise is a stable core (messages, tools, streaming, usage) with an explicit, namespaced extension field, and a capability matrix the router reads so it never sends a vision request to a text-only model. Token accounting deserves care: a tokenizer difference means the same 2,000-word document may be a different number of tokens per provider, so a per-tenant token quota should either be expressed in a provider-neutral unit, or enforced against the provider-reported usage after the call with a conservative pre-estimate before it. Rate limits are per provider key and often per model; a gateway that only enforces its own per-tenant limits still gets 429s from the provider when the aggregate exceeds the key's ceiling, so it needs both views and should shed or queue by priority before the provider does it for you. Nobody should trust static tables in a document like this for current limits; read the provider docs and expose them as configuration.
# Python 3 stdlib: sticky routing by rendezvous (highest-random-weight) hashing
import hashlib
def score(conv, backend):
return int(hashlib.sha256(("%s|%s" % (conv, backend)).encode()).hexdigest(), 16)
def pick(conv, backends):
return max(backends, key=lambda b: score(conv, b))
convs = ["conv-%d" % i for i in range(1000)]
full = ["cheap", "mid", "strong"]
before = {c: pick(c, full) for c in convs}
after = {c: pick(c, ["mid", "strong"]) for c in convs} # "cheap" removed (outage)
moved = sum(1 for c in convs if before[c] != after[c])
cheap_before = sum(1 for c in convs if before[c] == "cheap")
print("conversations pinned to cheap before:", cheap_before)
print("conversations that moved when cheap left:", moved)
print("moved but were NOT on cheap:", sum(1 for c in convs if before[c] != after[c] and before[c] != "cheap"))
# naive modulo hashing for contrast
naive = lambda c, n: int(hashlib.sha256(c.encode()).hexdigest(), 16) % n
moved_naive = sum(1 for c in convs if naive(c, 3) != naive(c, 2))
print("modulo hashing, 3 -> 2 backends, conversations moved:", moved_naive)
# Real output from python3:
# conversations pinned to cheap before: 333
# conversations that moved when cheap left: 333
# moved but were NOT on cheap: 0
# modulo hashing, 3 -> 2 backends, conversations moved: 654Stickiness exists because state lives on the provider side. Provider prompt caches (where offered) generally reuse a computed prefix only when the same prefix is sent to the same provider, often the same account or region, within a time window, so routing turns of one conversation to different providers throws away cached prefill and raises both cost and TTFT. Conversation-level features (threads, server-side assistant state) can be even harder to move. Hash the conversation or tenant key to a preferred backend, and fall back only when that backend is unhealthy. The demo uses rendezvous hashing: removing one backend moves only the conversations pinned to it (333 of 1,000), whereas naive modulo hashing reshuffles most of them (654 of 1,000 when going from 3 to 2). Stickiness fights the load-balancing goal, since hot conversations can skew load, so it is a preference with a health override, not a hard pin.
Where do caches sit relative to routing?
request -> [exact cache: hash(model, params, messages)] -> hit? return
-> [semantic cache: embed(query) nearest >= threshold, SAME tenant + model family]
-> router -> provider
Streaming through the gateway:
first token not yet sent -> failover to another provider is still invisible to the client
after first token sent -> cannot silently switch: text would change mid-answer
-> forward an in-band error, or let the client resume/restart
Secrets:
provider keys in a secrets manager, injected at runtime, rotated, never logged or sent to clients;
callers get gateway-issued virtual keys (per team, revocable, budget-capped); redact prompts
and keys in logs; per-tenant "bring your own key" needs its own isolation.Cache placement is a correctness question as much as a cost one. An exact-match cache is safe and cheap; a semantic cache (see Semantic Caching for LLMs) can return an answer written for a similar but different question, so it must be scoped by tenant and model family, gated by a similarity threshold you evaluated, and skipped for personalised or sensitive requests. Put both before the router so hits cost no provider call, and log hit and miss as first-class metrics. Streaming changes the failover story: until the first byte reaches the client the gateway can switch providers freely, but after that a switch would splice two different generations, so the gateway forwards an error frame and relies on client-side resume or restart as described in Streaming Responses, and it must relay frames without buffering. Secrets handling is plain but often asked: provider keys live only in the gateway's secret store, teams receive virtual keys that can be revoked and budget-limited (also the basis of cost attribution), and prompts and completions are treated as potentially sensitive in logs (see Authentication and Authorization).
This simulation uses a virtual clock (no sleeping), three providers with different price units, latencies and failure rates, and a hard outage of the cheapest one from t=30 s to t=60 s. Each call's randomness depends only on provider, request and attempt, so all four configurations face identical luck and the comparison is fair. Prices are made-up units, not real prices.
# Python 3 stdlib: multi-provider router with retries, fallback chain and a circuit breaker.
# Virtual clock (no sleeping), seeded. Costs are made-up UNITS, not real prices.
import random
TIMEOUT = 2.0 # seconds a hung call costs us
class Provider:
def __init__(self, name, cost, mean_lat, fail_rate, outage=None):
self.name, self.cost, self.mean_lat = name, cost, mean_lat
self.fail_rate, self.outage = fail_rate, outage
def call(self, t, key):
"""returns (ok, latency_seconds); draws depend only on (provider, request, attempt)
so every configuration below sees identical 'luck' and the comparison is fair"""
rng = random.Random("%s-%s" % (self.name, key))
if self.outage and self.outage[0] <= t < self.outage[1]:
return False, TIMEOUT # provider down: calls hang until timeout
if rng.random() < self.fail_rate:
return False, 0.05 # fast 5xx / 429
return True, rng.expovariate(1 / self.mean_lat)
class Breaker:
def __init__(self, threshold=5, cooldown=10.0):
self.threshold, self.cooldown = threshold, cooldown
self.fails, self.opened_at = 0, None
def allow(self, t):
if self.opened_at is None: return True
if t - self.opened_at >= self.cooldown: # half-open: let a probe through
self.opened_at = t
return True
return False
def record(self, ok, t):
if ok: self.fails, self.opened_at = 0, None
else:
self.fails += 1
if self.fails >= self.threshold: self.opened_at = t
def make_providers():
return [Provider("cheap", 1, 0.6, 0.05, outage=(30, 60)), # cheapest, has an outage
Provider("mid", 3, 0.9, 0.02),
Provider("strong", 10, 1.5, 0.01)]
def run(name, retries, fallback, use_breaker, n=300, dt=0.33):
provs = make_providers()
chain = provs if fallback else provs[:1] # policy: cheapest first
br = {p.name: Breaker() for p in provs}
rng = random.Random(42)
lats, ok_count, cost, attempts, wasted_outage, share = [], 0, 0, 0, 0, {}
for i in range(n):
t0 = i * dt; t = t0; done = False
for p in chain:
if use_breaker and not br[p.name].allow(t):
continue # skip an open circuit instantly
for a in range(1 + retries):
ok, lat = p.call(t, (i, a))
attempts += 1
if p.outage and p.outage[0] <= t < p.outage[1]: wasted_outage += 1
t += lat
br[p.name].record(ok, t)
if ok:
done = True; cost += p.cost; share[p.name] = share.get(p.name, 0) + 1; break
t += 0.1 * (2 ** a) * rng.uniform(0.5, 1.0) # backoff with jitter
if t - t0 > 6.0: break # overall deadline
if done or t - t0 > 6.0: break
ok_count += done
lats.append(t - t0)
lats.sort()
print("%-28s ok=%5.1f%% p50=%.2fs p95=%.2fs attempts=%3d calls-into-outage=%3d cost=%4d share=%s"
% (name, 100 * ok_count / n, lats[n // 2], lats[int(n * 0.95)], attempts, wasted_outage, cost, share))
run("A: cheapest only, no retry", retries=0, fallback=False, use_breaker=False)
run("B: retry x2, no fallback", retries=2, fallback=False, use_breaker=False)
run("C: retry x1 + fallback chain", retries=1, fallback=True, use_breaker=False)
run("D: C + circuit breaker", retries=1, fallback=True, use_breaker=True)
# Real output from python3:
# A: cheapest only, no retry ok= 67.0% p50=0.62s p95=2.09s attempts=300 calls-into-outage= 91 cost= 201 share={'cheap': 201}
# B: retry x2, no fallback ok= 73.7% p50=0.68s p95=6.59s attempts=486 calls-into-outage=255 cost= 221 share={'cheap': 221}
# C: retry x1 + fallback chain ok=100.0% p50=0.68s p95=5.79s attempts=488 calls-into-outage=176 cost= 474 share={'cheap': 213, 'mid': 87}
# D: C + circuit breaker ok=100.0% p50=0.46s p95=2.38s attempts=322 calls-into-outage= 11 cost= 478 share={'cheap': 211, 'mid': 89}
# Reading the output:
# - A: 33% of requests fail; the outage window alone accounts for about a third of the traffic.
# - B: retrying a hard-down provider adds timeouts (p95 6.59 s) and 255 wasted calls into the outage.
# - C: fallback gives 100% success but 176 calls still go into the outage, so p95 stays 5.79 s.
# - D: the breaker cuts calls into the outage from 176 to 11 and p95 to 2.38 s; cost is about 1% higher than C.
# - p50 also drops in D because ~30% of requests were inside the outage window in A-C.Not run here (needs a real gateway and provider accounts): a typical routing configuration expressed as data, shown only to illustrate shape.
# pseudo-config, not a real product's syntax, no output shown
routes:
- match: {task: summarize}
candidates:
- {provider: cheap, model: small-v1, weight: 90}
- {provider: mid, model: medium-v2, weight: 10} # canary
fallback: [mid/medium-v2, strong/large-v3]
retry: {max: 1, backoff_ms: 100, retry_on: [timeout, 429, 5xx]}
breaker: {consecutive_failures: 5, cooldown_s: 10}
sticky_key: conversation_id
cache: {exact: true, semantic: {enabled: false}}Q: Why put a gateway in front of LLM providers instead of calling them directly?
A: Centralisation: one place for auth, quotas, retries, fallback, logging and cost attribution instead of every service reimplementing them and holding provider keys. It also lets you change providers or models without changing applications, and it enforces budgets so one runaway loop cannot exhaust the account. The cost is an extra hop (typically a few milliseconds to tens of milliseconds, small next to seconds of model latency) and another tier that must be as reliable as the providers it protects.
Q: How would you design the routing policy?
A: Filter first, optimise second: drop candidates that cannot do the task (missing tool support, context too short, tenant not allowed) or are unhealthy or out of quota, then pick among the rest by cost, latency or a quality tier, and keep the remaining candidates as the fallback chain. Add weighted canary traffic for new models with per-variant metrics and a kill switch. I would avoid pure lowest-latency routing without damping because it herds traffic and oscillates.
Q: A provider has a 30-minute outage. What happens in your system?
A: The first failures are timeouts or 5xx; retries are limited and budgeted, and after N consecutive failures the circuit opens so the gateway stops calling that provider and sends traffic straight to the next candidate. A periodic half-open probe detects recovery. The trade-offs are higher cost and slightly different answers from the fallback model, which is why it must be pre-validated, and load on the fallback provider, which needs capacity or its own rate-limit headroom for the redirected traffic.
Q: How do you handle different token counts across providers?
A: The same text tokenizes differently per provider, so I do not use one tokenizer for everything. Before the call I use a conservative estimate for context-length checks and pre-quota, and after the call I record the provider-returned usage as the source of truth for billing and quota reconciliation. Cross-provider budgets are stored in money or a neutral unit, not raw tokens.
Q: Why does sticky routing matter for LLMs?
A: Provider-side prompt caching generally works only when the same prefix goes to the same provider (often same account and region) soon enough, so bouncing a conversation between providers repeats prefill for a long system prompt or history, raising cost and TTFT. I hash the conversation id to a preferred provider and override on unhealthy or saturated targets. The trade-off is uneven load, since a few very long conversations can skew a backend.
Q: Where does a semantic cache go, and what can go wrong?
A: Before the router, after auth, so hits skip the provider entirely, and keyed by tenant and model family. It can return an answer for a similar-but-different question, leak content across tenants if the scope is wrong, or serve stale facts, so it needs a tuned similarity threshold, TTLs, a bypass for personalised or sensitive prompts, and hit-quality monitoring. Exact-match caching is the safe first step.
Q: How does streaming change failover?
A: Before the first token has reached the client, failover is invisible, so I fail over on connection errors, timeouts to first byte and overload responses. Once tokens have been sent I cannot switch providers without changing text the user already read, so the gateway emits an error event and lets the client resume or restart. The gateway also must relay frames without buffering, or it destroys the time-to-first-token benefit.
Follow-up: Now the gateway is a single point of failure. How do you avoid that?
A: Run it as stateless replicas across zones behind a load balancer, keep configuration cached locally so a config-store outage does not stop traffic, and make shared dependencies (rate-limit counters, caches) fail open or to conservative local limits instead of blocking. Deploy it progressively like any critical service and provide a bypass or fallback path for the most critical calls. If availability of a single provider is 99.5% and two were independent, combined availability would be about 99.9975%, but outages are correlated (shared cloud, shared upstream), so treat that figure as an upper bound; and the gateway's own availability multiplies in.
Follow-up: How do you prevent retry storms across layers?
A: Decide which layer retries, normally the gateway, and have others pass a hint (a header or context flag) not to retry. Use exponential backoff with jitter, a per-provider retry budget (for example allow retries up to a small percentage of recent successful traffic), and an overall deadline propagated downstream. Honour Retry-After on 429s. Without these, three layers with three attempts each turn one request into 27, which prolongs the outage they are reacting to.
Follow-up: How do you attribute cost to teams and features when providers bill in aggregate?
A: Tag every request at the gateway with tenant, team, feature and prompt-version identifiers from the virtual key and headers, record provider-returned input and output token usage, and multiply by a versioned price table you maintain (prices change, so it is configuration with effective dates). Reconcile the sum against the provider invoice periodically and investigate the gap: retries, cancelled streams and cached tokens are the usual sources.
In any design that uses LLMs at scale, draw the gateway as its own tier and say what it enforces. Estimate with stated assumptions: ILLUSTRATIVELY, 5 million requests per day averages about 58 requests per second (5,000,000 / 86,400), with a peak of perhaps 3 to 5 times that, so a handful of stateless gateway replicas handle the routing logic easily; the limiting resources are the providers' rate limits and, for streams, concurrent connections, since each long-lived stream holds a socket for seconds (about 230 concurrent streams at the average rate if each lasts 4 s, by Little's law: 58 x 4). State the cost lever numerically too: if an easy tier that is 60% of traffic can go to a model priced at one tenth, the blended cost falls to about 0.6 x 0.1 + 0.4 x 1 = 0.46 of the all-strong baseline (illustrative), which is why routing by difficulty is the biggest cost knob, ahead of micro-optimising prompts. Then cover resilience (breaker, fallback with validated alternatives, capacity reserved on the fallback provider), sticky routing for prompt caches, cache placement, secrets and virtual keys, and observability (tokens, TTFT, error class, provider, cache hit rate, per-tenant spend). See Design an LLM API Gateway for the complete walkthrough.
A gateway trades directness and simplicity for control: an extra hop and an extra critical service in exchange for reliability, governance and freedom to change providers. Multi-provider routing buys resilience and price leverage, but every added provider is more surface to evaluate, secure, monitor and reconcile, and fallbacks can produce inconsistent behaviour. A canonical schema reduces coupling but hides provider-specific features unless you build escape hatches. Sticky routing saves cost and latency through provider caches but works against even load, and aggressive caching saves money at the risk of stale or wrong answers. A gateway is the wrong choice for a prototype with one model and one team (a direct SDK call with timeouts is enough), for latency-critical paths where an extra hop is unacceptable and a sidecar or library is a better home for the same logic, or when the organisation will not staff it as production infrastructure, since an under-maintained gateway is worse than none.