Design an LLM API Gateway / Multi-Tenant AI Platform

An LLM API gateway is one internal endpoint that sits in front of several model providers (hosted APIs and self-hosted servers) and gives every team a single contract for authentication, quotas, routing, caching, logging and cost reporting. Interviewers like this question because it is a normal API-gateway problem with three twists that break the textbook answer: cost and capacity are measured in tokens, not requests; the response is a long-lived stream that may fail halfway; and the payloads (prompts) are often the most sensitive data the company handles. For a Backend or AI engineer it is also the most realistic "platform" design you will actually be asked to build, so the goal below is a complete answer: requirements, numbers, architecture, data model, the hard parts in depth, failure modes, and a scaling path.

Related pages: LLM gateway and multi-provider routing, rate limiting and throttling, semantic caching for LLMs, streaming responses, and fault tolerance and circuit breakers. This page is the end-to-end case study that ties them together.

What Is an LLM API Gateway?

An LLM API gateway is a single front door that every team in a company goes through to use AI models, instead of each team calling OpenAI, Anthropic, or other providers directly with their own keys. It's the layer that checks who's asking, makes sure no one is overspending, decides which underlying model actually handles the request, and keeps a record of what was asked and what it cost. Any team building an AI feature just talks to this one gateway and never has to worry about the provider behind it.

Where This Is Used

• Large companies with dozens of internal teams building AI features, where a central platform team runs one gateway so no team manages its own provider keys or billing.
• Startups like Portkey and OpenRouter that sell exactly this as a product: one API key that routes to many different AI model providers.
• Enterprises that need to guarantee sensitive prompts (like customer data or internal documents) get logged and filtered consistently, no matter which team or app sent them.
• Companies that want automatic fallback: if one AI provider goes down or is slow, the gateway quietly switches requests to a backup provider without the app noticing.

A Day-to-Day Example

Think of a large company's front reception desk. Instead of every employee, vendor, and visitor being handed a master key to wander the building, everyone checks in at reception first. Reception verifies who they are, logs the visit, decides which floor or department they should be sent to, and keeps the building secure and accountable. An LLM API gateway plays the same role for AI requests moving through a company.

Requirements

State these out loud before drawing anything. All numbers are ILLUSTRATIVE assumptions to be confirmed with the interviewer, not facts about any real platform.

FUNCTIONAL
  F1  One API (chat/completions-style + embeddings) for many models and providers;
      callers use a model ALIAS ("chat-large"), not a vendor model id.
  F2  Authenticate each caller, resolve tenant + plan; per-tenant model allow-list.
  F3  Rate limits and budgets per tenant / per key / per model, counted in TOKENS.
  F4  Routing with weights, health awareness and automatic fallback between providers.
  F5  Streaming (SSE) and non-streaming, same contract for every provider.
  F6  Exact-match cache and (opt-in) semantic cache.
  F7  Usage + cost attribution per tenant / team / project; exportable for chargeback.
  F8  Request/response logging with PII controls, per-tenant retention settings.
  F9  Provider secrets never exposed to callers; rotation without downtime.
  F10 Config (routes, limits, prices) changeable without redeploying the gateway.

NON-FUNCTIONAL (assumed targets)
  N1  Gateway-added latency: p50 < 5 ms, p99 < 20 ms (excluding provider time).
  N2  Availability: 99.95% for the gateway itself (it is in every AI call's critical path).
  N3  Correctness of quotas: over-admission bounded (e.g. within a few %), never unbounded.
  N4  Streaming: first token forwarded as soon as the provider emits it (no buffering).
  N5  No cross-tenant data leakage (cache, logs, errors).
  N6  Provider outage must degrade to fallback or a clean 503, not to a hang.
OUT OF SCOPE (say so): model training/fine-tuning, prompt authoring UI, the RAG
pipeline behind individual apps, billing invoices/payments (we only produce the usage ledger).

Capacity Estimation

Estimation shows you know which dimension is the bottleneck. Here the surprise is that request rate is small and cheap, while concurrent open streams, token volume and log volume are what size the system.

ASSUMPTIONS (ILLUSTRATIVE)
  50M requests/day, average 1,200 input tokens, 400 output tokens
  peak-to-average ratio 3x, average streamed response lasts 8 s
  ~4 bytes of text per token, 25% of requests hit the exact cache

TRAFFIC
  average rps  = 50,000,000 / 86,400              ~ 580 rps
  peak rps     = 580 x 3                          ~ 1,750 rps
  in-flight streams at peak (Little's law: L = lambda x W)
               = 1,750 x 8 s                      ~ 14,000 open connections

TOKENS (drives provider spend and provider rate limits)
  input  tokens/day = 50M x 1,200 = 60B      output tokens/day = 50M x 400 = 20B
  spend/day = (60B / 1M) x input_price + (20B / 1M) x output_price
              -> use CURRENT per-million-token prices from provider docs; do not memorise

GATEWAY NODES
  async I/O proxy, assume ~2,000 concurrent streams per node (mostly idle waiting on the provider)
  peak: 14,000 / 2,000 = 7 nodes;  run N+2 across 3 AZs -> ~10-12 nodes (small instances)

RATE-LIMIT STORE (Redis)
  ~2 ops per request (reserve + reconcile) x 1,750 rps ~ 3,500 ops/s  -> trivial for one primary
  the real risk is one HOT tenant key, not aggregate throughput

USAGE / LOG VOLUME
  usage event ~ 0.5 KB x 50M = 25 GB/day (always kept, then aggregated)
  full bodies ~ (1,600 tokens x 4 B) = 6.4 KB x 50M = ~320 GB/day raw if logged for everyone
  -> bodies are OFF by default / sampled / per-tenant opt-in; retention measured in days, not years

CACHE
  25% exact hits -> 25% fewer provider calls and near-zero latency for those requests
  (real hit rates are workload-specific: measure before promising savings)

Two conclusions follow. First, the gateway is I/O-bound and cheap: a handful of nodes suffice, so the design effort goes into correctness (quotas, failure handling, privacy) rather than raw throughput. Second, the provider bill dwarfs the gateway's own infrastructure cost, which is why caching, routing to cheaper models and hard budgets earn their complexity.

High-Level Architecture

  Clients: apps, agents, batch jobs, internal tools
           | HTTPS (REST + SSE)
           v
  +--------------------------------------------------------------+
  |  L7 load balancer  (multi-AZ, TLS, health checks, no buffering)|
  +--------------------------------------------------------------+
           |
           v
  +--------------------------------------------------------------+
  |  GATEWAY  (stateless replicas, async I/O)                     |
  |   1 authenticate  API key / JWT -> tenant, plan, scopes       |
  |   2 policy        model allow-list, max_tokens cap, size limit |
  |   3 admit         token bucket + budget: RESERVE tokens        |
  |   4 cache         exact  ->  semantic (opt-in)                |
  |   5 route         alias -> candidates, weights, health         |
  |   6 adapter       normalise request/response, pass stream      |
  |   7 finish        RECONCILE tokens, emit usage event, log       |
  +---+-----------+------------+-------------+------------------+
      |           |            |             |
      v           v            v             v
  +--------+  +--------+  +-----------+  +-----------------------+
  | Redis  |  | Config |  | Caches    |  | PROVIDER POOL         |
  | limits |  | store  |  | exact:    |  |  hosted provider A    |
  | budget |  | (ver-  |  |  Redis    |  |  hosted provider B    |
  | breaker|  | sioned)|  | semantic: |  |  self-hosted (vLLM/..)|
  +--------+  +--------+  |  vector ix|  +-----------------------+
                          +-----------+
      |  async (never on the request path)
      v
  Kafka / PubSub -> usage ledger (OLAP)  |  redacted log store  |  metrics + traces -> alerts

  Control plane (separate): admin API/UI -> validate -> config store -> staged rollout to gateways
  Secrets: KMS / secret manager -> loaded into gateway memory, never logged

The split that matters is data plane versus control plane. The data plane (the gateway replicas on the hot path) only reads configuration and only writes counters and events; it can keep serving from its last-known-good config if the control plane or config store is down. Everything slow, batch-like or human-driven (dashboards, price updates, key management) lives in the control plane so that its failures cannot take down inference traffic.

Data Model

-- relational store (Postgres-style): small, slow-changing, read via cache/snapshots
tenants(id, name, plan_id, status, data_region, retention_days, log_bodies BOOL, created_at)
api_keys(id, tenant_id, key_prefix, key_hash, scopes[], expires_at, revoked_at)
        -- store a hash of the key (keys are high entropy random), show the secret once
plans(id, rpm, tpm, max_concurrency, monthly_budget_usd, allowed_aliases[])
model_aliases(alias, capability_flags, max_context, default_params)
routes(alias, provider_id, provider_model, weight, priority, region, enabled)
providers(id, kind, base_url, secret_ref, timeout_ms, rpm_cap, tpm_cap)   -- secret_ref, NOT the key
price_table(provider_id, provider_model, in_per_mtok, out_per_mtok, effective_from)  -- versioned
config_versions(version, checksum, author, created_at, state)            -- rollout + rollback
audit_log(actor, action, object, before, after, at)

-- hot state (Redis): counters with TTL
bucket:{tenant}:{key}:tpm      ->  level, last_refill_ts
inflight:{tenant}              ->  concurrent request count
budget:{tenant}:{yyyymm}       ->  spent micro-USD (INCRBY)
breaker:{provider}:{model}     ->  state, failures, open_until
cache:x:{hash}                 ->  response blob (TTL)

-- append-only usage event (Kafka -> OLAP table partitioned by day)
usage_events(request_id, ts, tenant_id, team, project, alias, provider, provider_model,
             in_tokens, out_tokens, cached BOOL, status, ttft_ms, latency_ms,
             fallback_count, cost_micro_usd, price_version)

Note what is deliberately absent: the request and response bodies are not in the usage event, and cost is stored with the price version that produced it so a later price change never rewrites history. Bodies, when logged at all, go to a separate encrypted store with its own retention and access controls.

Deep Dive: Token-Based Rate Limiting, Budgets and Fairness

Why requests-per-minute alone fails
  tenant X: 100 requests/min x 200 tokens   =  20,000 tokens/min
  tenant Y: 100 requests/min x 20,000 tokens = 2,000,000 tokens/min   (same RPM, 100x the cost)

Reserve-then-reconcile (the output length is unknown at admission)
  admit:     cost_est = prompt_tokens_est + max_tokens          (worst case)
             bucket.try_reserve(cost_est)  else 429 + Retry-After
  finish:    actual = provider usage (in + out)
             bucket.refund(cost_est - actual)                    (give unused tokens back)
  failure:   refund everything (a failed call should not eat quota)

Layers, checked cheapest-first:
  1. per-key concurrency (in-flight streams)      2. per-key RPM (abuse / bug guard)
  3. per-tenant TPM (token bucket)                4. per-tenant monthly budget (money)
  5. GLOBAL outbound limiter per provider key     (so tenants share upstream quota fairly)

A token bucket measured in tokens is the natural primitive: capacity is the allowed burst, the refill rate is the sustained tokens per second, and a request "pays" its estimated cost up front. You must reserve the worst case (max_tokens) because you do not yet know the completion length, then reconcile against the provider's reported usage when the response finishes; skipping the reserve lets a burst of long generations overshoot the limit before any usage is known. Prompt tokens have to be estimated at the gateway, and the tokenizer differs per provider, so the estimate is deliberately conservative (for example characters divided by 3.5 or 4) and is corrected by the provider's actual usage afterwards. Also cap max_tokens by plan, otherwise a caller can reserve and hold a huge budget.

Where it breaks at scale: the check must be atomic. In Redis this is a single Lua script (or a single-key transaction) that refills, compares and subtracts in one round trip; two separate GET/SET calls race and over-admit. A single very hot tenant then concentrates all its traffic on one key on one shard, so aggregate 3,500 ops/s is fine while one tenant at 2,000 ops/s can saturate a shard. The usual answer is to let each gateway node lease a slice of tokens (say a few hundred milliseconds of budget) and sync in the background, which cuts Redis calls by an order of magnitude at the price of bounded over-admission (each node can spend its slice even if the global bucket just emptied). Decide the failure policy for the limiter store explicitly: if Redis is unreachable, falling open protects availability but removes the cost guard, while falling closed protects money but turns a cache outage into a platform outage. A common compromise is falling back to a conservative local per-node limit. Fairness across tenants when the provider itself rate-limits you (tokens-per-minute caps on your provider key) needs a global outbound limiter and weighted fair queuing by plan, or one noisy tenant starves the rest. See rate limiting and throttling for the algorithms and Redis and caching for the store.

Deep Dive: Routing, Fallback and Retries Without Retry Storms

alias "chat-large" (config, versioned)
  candidates (ordered by priority, then weighted within a priority tier):
     tier 1:  provider-A/model-x   weight 70   region=us
              provider-B/model-y   weight 30   region=us
     tier 2:  self-hosted/model-z  weight 100  region=us      (fallback only)
  filters BEFORE weighting: capability (tools, vision), context window >= prompt,
                            data-region compatibility, provider not breaker-open

request path
  pick from tier 1 by weight  ->  call  ->  ok?  done
        |                              |
        |                        retryable? (429, 5xx, timeout, reset BEFORE first byte)
        |                              | yes, and retry budget allows
        +-------- next candidate  <----+
  non-retryable (400 bad request, content-policy refusal): return to caller, do not fan out

retry budget: retries may add at most ~10% extra load over a window; otherwise fail fast
breaker (per provider+model): closed -> open after N failures / high error rate
                              open -> half-open probe after cooldown -> closed

Provider outages, throttling (HTTP 429) and latency spikes are routine, so fallback is a first-class path, not an exception handler. Weighted routing spreads load (and hedges vendor risk); health-aware routing removes a provider quickly using a circuit breaker fed by passive signals (error rate and latency EWMA per provider and model) plus optional active probes. Only some failures justify trying another provider: 429, 5xx, timeouts and connection errors before the first response byte are retryable; a 400 or a policy refusal will fail identically everywhere and only multiplies cost. Retries need a budget, because during a partial outage every gateway node retrying every failed call multiplies load on the survivors and turns a brownout into a full outage (the retry storm covered in fault tolerance and circuit breakers).

Fallback is not free of semantics. A different model can answer differently, use a different tokenizer (so cost and context limits change), support different tool-calling schemas, and cost more, so the adapter layer normalises formats, the route must filter on capability and context length, and the caller should be able to opt out of cross-model fallback when reproducibility matters. Compliance can forbid fallback entirely: a tenant pinned to an EU region must not silently spill into a US provider. Hedged requests (sending a second call if the first is slow) cut tail latency but double the spend on those calls, so they are usually limited to short, cheap, latency-critical routes. For the routing policy space and provider adapters see LLM gateway and multi-provider routing.

Deep Dive: Streaming Pass-Through and Mid-Stream Failure

client <--SSE-- gateway <--SSE/HTTP chunked-- provider

gateway per-stream loop (async, no full-response buffer):
   read chunk from provider -> normalise event shape -> write to client -> flush
   on client disconnect      -> CANCEL the upstream request (else you pay for unread tokens)
   on final usage chunk      -> capture tokens (some providers need an option to include usage)

timeouts (all four, each different):  connect | time-to-first-byte | idle gap between chunks | total
failure timeline
   [--- before first byte ---]   safe: retry / fall back, client sees nothing
   [--- after first token  ---]   NOT safe to switch: partial text already shown
                                  -> send an error event / close cleanly, let the client retry
                                     (or resume only where the app can tolerate a re-generation)

The gateway must be a streaming proxy, not a request-response wrapper: buffering the whole completion destroys time-to-first-token, and reverse proxies or load balancers in the path often buffer by default, so disable response buffering for these routes. Concurrency is the sizing constraint (Little's law above gave ~14,000 open connections), which is why the proxy is async and why per-key concurrency limits exist. Two failure details separate good answers from average ones. First, client disconnects must propagate upstream as cancellations; otherwise abandoned tabs keep generating tokens you are billed for. Second, once the first token has been forwarded you generally cannot transparently fail over, because the client has already rendered partial output and another model would produce a different continuation; the honest options are to end the stream with an error event and let the client retry, or, for idempotent use cases, restart the generation and mark the response as regenerated.

Token accounting for streams comes from the provider's final usage information when it is available (this varies by provider, and some need it enabled in the request); when it is missing, count chunks or run a tokenizer as an estimate and label it as such in the ledger. Output guardrails on a stream force a trade-off: checking each chunk lets you cut a bad stream immediately but can miss patterns spanning chunks, while buffering a window catches more but adds latency. Deploys also matter: rolling out a new gateway version must drain in-flight streams (stop accepting new ones, allow long ones to finish up to a limit) or every release cuts thousands of responses. See streaming responses.

Deep Dive: Exact and Semantic Caching in a Multi-Tenant Gateway

EXACT CACHE
  key = sha256( tenant_scope | alias-or-provider_model | canonical(messages, system, tools)
                | temperature, top_p, max_tokens, seed | prompt_template_version )
  eligible: temperature == 0 (or caller opts in), no tool side-effects, not personalised/time-sensitive
  hit: replay stored response (re-emit as SSE chunks if the caller asked for a stream)
  concurrent identical misses: single-flight so one provider call serves all

SEMANTIC CACHE (opt-in, per tenant + per alias)
  embed(query) -> ANN search among cached queries in the SAME scope
  if cosine_sim >= threshold (tuned on labelled pairs) -> return cached answer
  danger: "cancel order 123" vs "cancel order 124" embed almost identically

The exact cache is cheap and safe when the key contains everything that influences the output: model, full canonicalised messages, sampling parameters and the prompt template version, plus the tenant scope so one tenant's prompts and answers can never be returned to another (the classic multi-tenant cache bug is a key without the tenant). It only makes sense for deterministic-ish requests; with temperature above zero a cached answer removes the variety the caller asked for, so caching should be explicit. Streaming callers still benefit: the gateway replays the stored text as chunks, which also makes cached responses cost no tokens against the tenant's quota.

The semantic cache widens the hit rate by matching paraphrases, but it introduces a correctness risk that exact caching does not have: a false positive returns a confident answer to a different question. Mitigations are a high, empirically tuned similarity threshold, scoping by tenant, model, system prompt and tool set, skipping it for personalised, transactional or freshness-sensitive routes, and measuring the false-hit rate on a labelled sample before enabling it. It also adds an embedding call and a vector lookup to every miss, so it must pay for itself in avoided provider calls. Provider-side prompt (prefix) caching, where a provider supports it, is a separate mechanism that discounts repeated prompt prefixes at the provider; a gateway can help it by keeping stable prompt prefixes and, where useful, routing the same tenant's related requests to the same backend, but the details are provider-specific, so check current documentation. Both cache layers must be invalidated by TTL and by model or template version changes. See semantic caching for LLMs.

Deep Dive: Secrets, Logging with PII Controls, Cost Attribution and Config Rollout

SECRETS
  provider keys: secret manager/KMS -> gateway memory (never in config repo, logs or error bodies)
  rotation: add new key (dual-active) -> shift traffic -> revoke old  (zero downtime)
  tenant keys: random, high entropy; store only a hash + prefix; scoped; revocable; expiry

LOGGING TIERS
  tier 0 (always)   metadata only: request_id, tenant, alias, provider, tokens, latency, status, cost
  tier 1 (opt-in)   redacted prompt/response bodies, encrypted at rest, short retention, audited access
  never             Authorization headers, raw provider keys, unredacted bodies of zero-retention tenants

CONFIG ROLLOUT
  edit -> schema + safety validation (weights sum, no route to disabled provider, limits sane)
       -> publish version N+1 -> canary on 1 node / 5% -> watch error rate, latency, 429s
       -> 25% -> 100%   |  any regression -> automatic rollback to N  |  nodes keep last-known-good

Prompts routinely contain customer data, credentials pasted by users and other regulated content, so logging is a design decision, not a default. Always keep cheap metadata (enough for cost, debugging and abuse detection); make body logging opt-in per tenant, redact before storage (pattern rules and ML-based detectors help but are imperfect, so treat redaction as risk reduction, not a guarantee), encrypt, set a short retention, and audit who reads logs. Some tenants need a zero-retention mode where nothing beyond counters is stored. Remember that the providers have their own data-handling terms, and the gateway is the place to enforce which tenants may use which provider.

Cost attribution depends on trustworthy usage events: emit them asynchronously through a queue so a slow analytics store never blocks a request, key them by request_id so replays are idempotent (the pipeline is at-least-once, so duplicates must be deduplicated, giving effectively-once counting), store the price version, and require caller-supplied tags (team, project, environment) so chargeback is possible. Reconcile the ledger periodically against the provider invoices; drift usually means missing cancelled-stream usage or retried calls being counted twice. Configuration is a classic outage source, because a bad routing weight or limit ships to every replica at once. Version and validate it, roll it out in stages with automatic rollback, and make gateways keep serving the last-known-good snapshot if the config store is unreachable. Watch the metrics that matter for tenants and providers: error and 429 rate, fallback rate, time-to-first-token, tokens per second and cache hit rate, with a request id propagated through traces (for example with OpenTelemetry) so one slow call can be followed end to end. See observability for AI systems and authentication and authorization.

Failure Modes

FAILURE                          DETECTION                     BEHAVIOUR / MITIGATION
provider A hard down             error rate, breaker opens     route to tier 2; alert; cap retries
provider A slow (brownout)       TTFT / latency EWMA           shift weight away; hedge only cheap routes
provider 429 (our upstream cap)  429 count per provider key    global outbound limiter, queue or shed low-priority tenants
Redis (limits) unavailable       client errors, timeouts       fail to local conservative limits; do NOT hang the request
config store down                fetch errors                  keep last-known-good snapshot; block config changes only
bad config pushed                canary metrics regress        automatic rollback to previous version
client disconnects mid-stream    socket close                  cancel upstream, reconcile partial usage, log status=client_abort
provider dies mid-stream         upstream reset after tokens   error event to client; no silent switch; count partial usage
retry storm                      retry ratio metric            retry budget + jittered backoff + breaker
noisy tenant floods              per-tenant 429 rate           per-key concurrency + TPM; weighted fair queuing upstream
duplicate/lost usage events      ledger vs provider invoice    dedupe on request_id; periodic reconciliation
cache poisoning / cross-tenant   audit, tests                  tenant scope in every key; never cache tool side-effects
key leak (tenant)                anomaly on usage spike        revoke by prefix, per-key budget alert
gateway node crash               LB health check               streams on that node fail; clients retry; N+2 capacity

Scaling Path

STAGE 1  one team, ~10 rps
  single gateway service (2 replicas), Postgres for config, Redis for counters,
  usage events written to a table, simple weighted routing + fallback.

STAGE 2  company-wide, ~600 rps average
  stateless replicas across 3 AZs behind an L7 LB, async usage pipeline (queue -> OLAP),
  exact cache in Redis, breaker per provider, staged config rollout, redacted logging tiers.

STAGE 3  ~2k+ rps peak, many tenants, strict SLAs
  token leasing per node to cut Redis calls, hot-tenant isolation, semantic cache (opt-in),
  global outbound limiter with fair queuing, priority classes (interactive vs batch).

STAGE 4  multi-region, regulated tenants
  regional gateways with regional quotas (approximate global limits, eventual reconciliation),
  data-residency routing (never fall back across regions when forbidden),
  cell-based deployment so one cell's failure or one tenant's spike has a bounded blast radius,
  dedicated capacity or dedicated provider keys for the largest tenants.

The scaling order follows where the pain shows up first: correctness and privacy before throughput, then the limiter store and hot tenants, then regional and isolation concerns. In an interview, volunteer this staging; it signals that you would not start with the multi-region cell architecture for a ten-rps internal tool.

How to Run the Interview (about 45 minutes)

• Minutes 0-5: clarify scope (internal platform or external product? streaming? regulated data? how many tenants?) and state assumptions and SLOs.
• Minutes 5-10: do the capacity estimate and say what it implies: request rate is small, open streams, tokens and logs are large.
• Minutes 10-20: draw the architecture (data plane, control plane, async usage path) and the data model; name what is on the hot path and what is not.
• Minutes 20-38: go deep where they push: token-based limiting (reserve and reconcile), fallback and retry budgets, mid-stream failure, cache correctness, logging and PII.
• Minutes 38-45: failure modes, the staged scaling path, and the trade-offs you consciously accepted (fail-open versus fail-closed limits, cross-model fallback semantics, cache staleness).

Key Pointers

• The gateway is I/O-bound and small; the sizing drivers are open streams (Little's law: rate x duration), token volume and log volume, not request rate.
• Rate-limit and budget in tokens, not requests: reserve the worst case (prompt estimate plus max_tokens) before calling, reconcile with the provider's real usage afterwards, and refund failures.
• The limiter check must be atomic (a Lua script or single-key transaction) and you must choose fail-open versus fail-closed for the limiter store explicitly; per-node token leasing trades exactness for fewer Redis calls.
• Fall back only on retryable failures (429, 5xx, timeouts, errors before the first byte), with a retry budget, jittered backoff and a circuit breaker; never fan out a 400 or a policy refusal.
• You can fail over freely before the first streamed byte; after the first token the honest options are an error event or a marked regeneration, not a silent switch. Cancel upstream when the client disconnects.
• Cache keys must include tenant scope, model, the canonical prompt, sampling parameters and template version; semantic caching adds false-positive risk and should be opt-in and tuned.
• Treat logging as tiered: metadata always, redacted bodies opt-in with short retention, never secrets; redaction reduces risk but is not a guarantee.
• Keep the data plane and control plane separate, roll out config in stages with automatic rollback, and let gateways serve last-known-good config when the control plane is down.

How It Works

1. The request reaches the load balancer, then a stateless gateway replica, which authenticates the API key or token and resolves the tenant, plan, allowed aliases and data region from an in-memory config snapshot.
2. Policy checks run first (allowed model alias, size limit, max_tokens cap), then the limiter reserves estimated tokens from the tenant's bucket and checks concurrency and monthly budget; a failure returns 429 with Retry-After before any provider cost is incurred.
3. The exact cache (and the semantic cache if enabled) is consulted; a hit is returned immediately, re-emitted as a stream if needed, and only counted as a cached request.
4. On a miss the router filters candidates by capability, context length, region and breaker state, picks one by weight, and the provider adapter converts the request into that provider's format.
5. The response is streamed back chunk by chunk; if the call fails before the first byte with a retryable error and the retry budget allows, the router tries the next candidate, otherwise the caller receives a clean error.
6. On completion the gateway reconciles token usage with the bucket, writes the response to the cache if eligible, and emits an asynchronous usage event (plus an optional redacted log) to the queue for the cost ledger and observability.

Try It Yourself

Standard-library Python only. It simulates 120 seconds of traffic from two tenants with a token-denominated bucket (reserve, then reconcile), an exact cache, weighted routing between two seeded providers, a tiny circuit breaker, and a fallback provider that is used only when the others fail. The primary provider is hard-down between t=40 s and t=60 s. All numbers are made-up simulation parameters.

import random, hashlib
random.seed(42)

class TokenBucket:                                   # capacity and cost are measured in TOKENS
    def __init__(self, capacity, refill_per_sec):
        self.capacity, self.rate = capacity, refill_per_sec
        self.level, self.last = float(capacity), 0.0
    def _refill(self, now):
        self.level = min(self.capacity, self.level + (now - self.last) * self.rate)
        self.last = now
    def try_reserve(self, cost, now):                # reserve worst case BEFORE calling the model
        self._refill(now)
        if self.level >= cost:
            self.level -= cost
            return True
        return False
    def refund(self, amount, now):                   # reconcile with real usage afterwards
        self._refill(now)
        self.level = min(self.capacity, self.level + amount)

class Provider:
    def __init__(self, name, weight, fail_rate, outage=None):
        self.name, self.weight, self.fail_rate, self.outage = name, weight, fail_rate, outage
        self.fails_in_row, self.open_until = 0, 0.0   # tiny circuit breaker
    def healthy(self, now):
        return now >= self.open_until
    def call(self, now):
        down = self.outage and self.outage[0] <= now < self.outage[1]
        ok = not down and random.random() > self.fail_rate
        if ok: self.fails_in_row = 0
        else:
            self.fails_in_row += 1
            if self.fails_in_row >= 3: self.open_until = now + 5.0   # open for 5 s
        return ok

providers = [Provider("primary",  70, 0.02, outage=(40, 60)),   # hard outage from t=40s to t=60s
             Provider("second",   30, 0.02),
             Provider("fallback",  0, 0.01)]                    # weight 0: only used on fallback
tenants = {"acme":  TokenBucket(capacity=60_000, refill_per_sec=9_000),
           "small": TokenBucket(capacity=8_000,  refill_per_sec=1_500)}
cache, stats = {}, {"cache_hit": 0, "throttled": 0, "ok": 0, "failed": 0, "fallbacks": 0}
served = {p.name: 0 for p in providers}
per_tenant = {t: {"ok": 0, "throttled": 0} for t in tenants}
prompts = [f"prompt-{i}" for i in range(1200)]        # pool size controls the exact-cache hit rate

def pick(now, exclude):
    pool = [p for p in providers if p.weight > 0 and p.healthy(now) and p not in exclude]
    return random.choices(pool, weights=[p.weight for p in pool])[0] if pool else None

for i in range(3000):
    now = i * 0.04                                    # 25 req/s of simulated time -> 120 s
    tenant = "acme" if random.random() < 0.7 else "small"
    prompt = random.choice(prompts)
    key = hashlib.sha256(("m1|" + prompt).encode()).hexdigest()
    if key in cache:                                  # exact cache: costs no provider tokens
        stats["cache_hit"] += 1; continue
    reserve = 600 + 400                               # 600 prompt + 400 max_tokens (assumed)
    if not tenants[tenant].try_reserve(reserve, now):
        stats["throttled"] += 1; per_tenant[tenant]["throttled"] += 1; continue
    tried, done = [], False
    while not done:
        p = pick(now, tried) or next((f for f in providers if f.weight == 0 and f not in tried), None)
        if p is None: break
        if tried: stats["fallbacks"] += 1
        tried.append(p)
        if p.call(now):
            served[p.name] += 1; done = True
    if done:
        stats["ok"] += 1; per_tenant[tenant]["ok"] += 1
        cache[key] = True
        tenants[tenant].refund(reserve - 650, now)    # actual usage assumed 650 tokens
    else:
        stats["failed"] += 1
        tenants[tenant].refund(reserve, now)          # never bill a failed request

print("requests: 3000 over 120 simulated seconds; primary hard-down 40s-60s")
print("stats      :", stats)
print("served by  :", served)
print("per tenant :", per_tenant)
# Real output from python3:
# requests: 3000 over 120 simulated seconds; primary hard-down 40s-60s
# stats      : {'cache_hit': 1743, 'throttled': 163, 'ok': 1094, 'failed': 0, 'fallbacks': 35}
# served by  : {'primary': 623, 'second': 468, 'fallback': 3}
# per tenant : {'acme': {'ok': 869, 'throttled': 0}, 'small': {'ok': 225, 'throttled': 163}}
# Reading it: 1,743 of 3,000 requests never reached a provider (exact cache).
# Tenant "small" has a smaller token bucket, so 163 of its calls were throttled while "acme" had none.
# 35 fallback hops were spread over the run (random 2% failures plus the outage); after 3 consecutive
# failures the breaker keeps the primary out of rotation for 5 s, so during the outage the second provider
# took most traffic (primary served 623 vs second 468 instead of the 70/30 split) and no request failed.
# The last-resort provider served only 3 requests.
# Not modelled: streaming, mid-stream failures, retry budgets, semantic cache, Redis, real latencies.

Interview Q&A

Q: Why not just let each team call the providers directly with their own keys?

A: You can for a prototype, but at company scale you lose cost visibility (who spent what), central rate limiting against shared provider quotas, one place to rotate secrets, a consistent logging and PII policy, and the ability to add a provider or fail over without changing every client. The price is a new critical dependency and an extra hop, which is why the gateway needs N+2 capacity, a low added-latency target (single-digit milliseconds at p50 in my assumptions) and a last-known-good config mode.

Q: Why count tokens instead of requests for rate limiting?

A: Provider capacity and money are consumed in tokens, so two tenants with the same request rate can differ in cost by 100x. I reserve prompt estimate plus max_tokens at admission because completion length is unknown, then refund the difference from the provider's reported usage. Keep a request-rate and concurrency limit as well, because those still protect the gateway itself and catch buggy loops.

Q: A provider starts returning 503s for 30% of calls. What happens in your design?

A: Passive health tracking raises the error rate, the circuit breaker opens for that provider and model, and the router shifts traffic to the remaining candidates or the fallback tier while a half-open probe checks for recovery. Retries are allowed only within a budget (say about 10% extra load) with jittered backoff, otherwise the survivors get overloaded. I would also alert on fallback rate, because fallback hides the outage from callers and can silently raise cost.

Q: How do you handle streaming when the provider fails halfway?

A: Before the first byte I can retry or fall back invisibly. After tokens have been forwarded I cannot switch models transparently, since the client already shows partial text and a different model would continue differently. I close the stream with an error event, record partial usage, and let the client retry; for idempotent workloads I can restart the generation and flag it as regenerated.

Q: How do you make the cache safe in a multi-tenant system?

A: Every key includes the tenant scope, the model, the canonicalised messages, the sampling parameters and the template version, so nothing from one tenant can be returned to another and a model or prompt upgrade invalidates naturally. Only deterministic or explicitly opted-in requests are cached, and single-flight prevents a stampede on identical concurrent misses. Semantic caching goes behind a tuned threshold and per-route opt-in because a false hit returns a confident wrong answer.

Q: How do you attribute cost to teams accurately?

A: Each request emits an immutable usage event with request id, tags (tenant, team, project), provider model, token counts, the price version and computed cost, sent through a queue. Because the queue is at-least-once I deduplicate on request id, giving effectively-once counting. I reconcile monthly against provider invoices, and drift usually points to cancelled streams or double-counted retries.

Q: What do you log, given prompts can contain PII and secrets?

A: Metadata always (no bodies), bodies only per-tenant opt-in or by sampling, redacted before storage, encrypted, short retention, and access audited; a zero-retention tenant setting stores nothing but counters. Redaction with patterns or ML detectors is imperfect, so I treat it as risk reduction and pair it with access controls and retention limits rather than promising it removes all sensitive data.

Q: How do you roll out a routing or limit change safely?

A: Config is versioned and validated (weights, limits, disabled providers), published to a small canary slice, compared on error rate, latency and 429 rate, then widened in stages with automatic rollback. Gateways keep the last-known-good snapshot if the config store is unreachable, so a control-plane failure cannot take down inference traffic.

Follow-up: Redis holding your rate-limit state goes down. Fail open or fail closed?

A: It depends on what the limiter protects. Failing open keeps the platform up but removes the cost guard, failing closed protects spend but converts a cache outage into a platform outage. I would fall back to conservative per-node local limits (for example a fraction of each tenant's normal rate divided by node count) plus the per-key budget caps enforced at the provider account level, and alert loudly so the over-admission window stays short.

Follow-up: One tenant has a huge burst and starves everyone else on the shared provider quota. How do you stop that?

A: Per-tenant token buckets stop a tenant from exceeding its own plan, but the shared upstream provider limit needs a second layer: a global outbound limiter per provider key with weighted fair queuing by plan and priority classes (interactive ahead of batch). Excess batch traffic is queued or shed with a 429, and very large tenants may get dedicated provider keys or capacity so their spikes have a bounded blast radius.

Follow-up: How would you keep a tenant's data inside one region while still having fallback?

A: Region becomes a hard filter on the candidate list before weights and fallback are applied, so a pinned tenant only ever sees providers and gateways in its region, and the fallback tier for that tenant is another in-region provider or a clean 503. The cost is lower resilience for those tenants, which I would call out explicitly, and the quota state, cache and logs must also stay regional.

System Design Tie-in

This case study appears whenever a company runs more than one AI feature: it is the shared "AI platform" tier under chat products, agents, batch enrichment and internal tools. The numbers an interviewer wants, as ESTIMATION with stated assumptions: roughly 50M requests a day is only about 600 rps average and under 2,000 rps at peak, but at an eight-second average stream that is around 14,000 concurrent connections, so the gateway is sized by open streams and async I/O rather than CPU, and about a dozen small nodes across three AZs is a reasonable ballpark. The rate-limit store sees a few thousand operations per second, which one Redis primary handles, so the hard limiter problems are atomicity, hot tenants and fail-open policy rather than throughput. Token volume (tens of billions of input tokens a day in this scenario) determines the provider bill, which is why caching, routing to cheaper models for simple tasks, and hard per-tenant budgets have a direct financial payoff; the exact dollar figure comes from the provider's current price sheet. Log volume is the surprise: full bodies at hundreds of gigabytes a day is why bodies are opt-in. The same building blocks recur elsewhere, in the rate limiter design, the cache tier and the load balancer, and this design leans on the multi-provider ideas in LLM gateway and multi-provider routing.

Trade-offs

A central gateway buys governance, cost control and provider independence at the cost of one more hop, one more critical dependency and a team that has to run it at higher availability than any single AI feature it serves. Token-based limiting is fairer and cheaper than request limiting but needs estimates and reconciliation, and leasing tokens per node reduces Redis load while allowing bounded over-admission. Automatic fallback improves availability but can change answer quality, tokenization and price, and can violate data-residency rules, so it should be policy-driven and observable rather than universal. Semantic caching raises hit rates but risks wrong answers, and full prompt logging aids debugging but creates a sensitive data store. It is the wrong choice for a single team with one provider and low volume (a shared client library with a budget alert is simpler), for workloads that need provider-specific features the normalised API cannot express, or when latency is so tight that even a few milliseconds and an extra network hop matter more than central control. Whatever you choose, keep the design honest about what it cannot do: it cannot make provider outputs deterministic, it cannot remove sensitive data from prompts with certainty, and it cannot prevent prompt injection by itself.