A load balancer spreads incoming requests across a pool of identical backends so that no single instance is the bottleneck and no single instance failing takes the service down. It looks like the easiest box on an architecture diagram, and interviewers probe it because the hard parts are hidden inside it: which signal the balancer uses to decide (request counts, connections, latency, queue depth), how quickly it notices a dead or slow backend, what happens to state that lives on one instance, and what happens when the balancer itself or its information is wrong. For a Backend or AI engineer the topic has a second layer, because LLM and GPU backends break the assumption classic algorithms rely on: that one request costs about as much as another. A 20-token completion and a 4,000-token completion are both "one request", and a balancer that counts requests will happily stack the long ones on the same GPU.
Load balancing is the practice of spreading incoming work across several identical servers instead of sending it all to one, so that no single machine gets overwhelmed and the whole system keeps running even if one server goes down. A dedicated piece of software or hardware, the load balancer, sits in front of the servers and decides, for every incoming request, which one should handle it. Any app or website that serves more people than one computer can handle needs some version of this, which is essentially all large-scale software.
Think of a supermarket with several checkout lanes and a staff member standing at the front directing customers: "you go to lane 3, you go to lane 5." That staff member is the load balancer — they glance at how long each line is and send new customers to whichever lane will get them through fastest, and if a lane's register breaks down, they stop sending anyone there until it's fixed. Without that person, everyone might pile into the same lane while three others sit empty.
L4 (transport) L7 (application)
client --TCP--> LB --TCP--> backend client --TLS/HTTP--> LB --HTTP--> backend
sees: src/dst IP, port terminates TLS, parses the request
picks a backend per CONNECTION sees: method, path, headers, cookies
forwards bytes (NAT or proxy) picks a backend per REQUEST
cheap, very high throughput path/host routing, retries, auth, rate limits
cannot see a request boundary more CPU and latency per request
HTTP/2 or gRPC through an L4 LB: ONE long-lived connection carries many requests,
so all of them land on whichever backend that connection was assigned to.An L4 balancer decides once per connection using only addressing information, so it is extremely cheap and protocol-agnostic, but it balances connections rather than work. That matters more than it sounds: with HTTP/2 or gRPC a client multiplexes many requests over one long-lived connection, so an L4 balancer can pin a very chatty client to one backend for hours while the others idle. An L7 balancer terminates the client connection, reads each request, and can send request 1 and request 2 from the same client to different backends, route by path (/v1/embeddings to one pool, /v1/chat to another), retry safely, and export per-request metrics. The price is more CPU, one extra hop of latency, and the balancer now holds a lot of state. See Load Balancing: L4 vs L7 for the networking detail and Reverse Proxy for the proxy role itself.
For LLM traffic the L7 choice is usually forced: you need to read the request to route by model name, to attach a deadline, and to count tokens. A common shape is an L4 or cloud network balancer at the edge for raw scale and availability, in front of an L7 layer (an API gateway, see LLM Gateway & Multi-Provider Routing) that makes the smart, request-aware decisions.
Algorithm Signal it needs Good when Weakness
------------------- -------------------------- ------------------------------ ---------------------------------
Round robin none uniform, short requests equalises COUNT, not WORK
Weighted RR static weights mixed instance sizes weights go stale
Random none many stateless LBs, no state variance: worst server gets ~log n / log log n
Least connections in-flight count per backend variable durations needs accurate state; herd on stale info
Power of two choices sample 2, pick fewer in-flight variable durations, many LBs slightly worse than global least-conn
Hash / consistent a key (user, session, model) cache or state affinity hot keys give hot backends
Latency-aware (EWMA) recent response time heterogeneous or degrading hosts latency ≠ load; can oscillate
Worked numbers (ILLUSTRATIVE): service time ~ Pareto(alpha = 1.5), scale 50 ms, so mean = 50 ms * 1.5/0.5 = 150 ms
uncapped (the demo caps at 20 s, giving ~142 ms) but a few requests take seconds. At 85% utilisation on 8 servers, round robin sends every 8th request to
the same server no matter what that server is doing; one unlucky run of long requests builds a queue there
while neighbours sit idle.Round robin is optimal only when requests cost about the same: it makes the request count equal, and with uniform costs the work follows. With heavy-tailed durations the work per server over any window is very noisy, and because round robin ignores what is happening it cannot correct: a server stuck on three slow requests keeps receiving a fair share of new ones, which then wait behind them. Least connections fixes this by using in-flight count as a proxy for queue length, so new work flows to whoever is free. Its own weakness is that it needs an accurate, global view: with several balancer instances each holding its own stale counts, they can all see the same backend as "least loaded" and stampede it (a herd effect).
Power of two choices (P2C) is the standard answer: pick two backends at random and send the request to the one with fewer in-flight requests. Sampling only two needs no global state, so it works with many balancers, and the classic result (Mitzenmacher, "the power of two choices") is that this small amount of choice removes most of the imbalance of pure random assignment: the maximum load drops from roughly log n / log log n to roughly log log n. Envoy's least-request balancer, for example, samples a small number of random hosts (two by default), and NGINX offers a random two least_conn mode. The demo further down measures all of these on the same seeded heavy-tailed workload.
hash(key) % N breaks when N changes: Consistent hashing ring (with virtual nodes)
8 servers -> 9 servers hash space 0 .. 2^128 wrapped into a ring;
key 1234: 1234 % 8 = 2 -> server 2 each server owns MANY points (s3#12, s0#7, ...)
key 1234: 1234 % 9 = 1 -> server 1 (moved!) key -> hash it, walk clockwise to the first
~ N/(N+1) of ALL keys change owner server point; that server owns the key
adding s8 inserts new points; it steals only the
Ring: adding a server moves only the keys in arcs just before those points, ~1/(N+1) of keys.
the arcs its new points cut out.Hash-based routing is for cases where the same key should reach the same backend: a cache shard, a per-user in-memory session, a per-document worker. Plain modulo hashing is fine until the pool size changes, and then almost every key moves, which for a cache means near-total cache loss (a stampede on the database) at exactly the moment you scaled up or lost a node. A consistent hash ring places each server at many pseudo-random points (virtual nodes) on a ring and assigns each key to the next point clockwise, so adding or removing one server moves only about 1/N of keys. Virtual nodes matter for balance: with one point per server the arcs are very uneven (the demo shows one server with 16 keys and another with 5,398 out of 20,000); with 100 points per server the spread is tight. The cost of any key-based scheme is hot keys: one viral document or one huge tenant means one hot backend, because hashing preserves skew rather than smoothing it. Variants such as "consistent hashing with bounded loads" cap how far above average a backend may go.
Sticky sessions (session affinity) are the same idea keyed on a cookie or source IP: the balancer remembers "this client goes to backend 3". They are a workaround for backends that keep per-user state in memory. The costs are real: load can skew (a few heavy users pin a backend), scaling in or a crash loses those sessions unless state is recoverable, and rolling deploys need draining. The generally preferred design is stateless backends with state in a shared store (Redis, a database), using stickiness only as an optimisation such as cache locality, never as a correctness requirement. See Cookies & Sessions.
Active check : LB probes GET /healthz every 5 s; mark DOWN after 3 consecutive failures
worst-case detection ~ 3 x 5 s = 15 s (ILLUSTRATIVE settings; tune per service)
Passive check : LB watches real traffic; eject a backend after e.g. 5 consecutive 5xx / timeouts
detects failure on real requests, but only after real users were hurt
liveness : "process is alive" -> restart it if false
readiness : "safe to send traffic now" -> add/remove from pool
(LLM server: false until weights are loaded and warmed up, which can take minutes)
Timeline of a backend dying at t=0 (5 s interval, 3 strikes):
t=0 backend dies; LB still routes to it
t=0..15 requests to it fail / time out -> retries land on healthy peers (if retry is enabled)
t=15 marked DOWN, traffic stops; connections in flight are broken
later probes succeed N times -> UP, ramped in with slow start, not full share instantlyA load balancer without health checks just spreads failure evenly. Active checks (synthetic probes) detect a dead backend even with no traffic; passive checks (outlier detection, used by Envoy and by many proxies) react to real errors and timeouts and catch backends that pass the probe but fail on real work. Use both, and make the probe meaningful: a /healthz that only returns 200 from the web framework will keep a GPU replica in rotation after its CUDA context has died. Detection time is a floor on your error window, so the retry policy carries the rest: retry idempotent requests on a different backend, with a per-request timeout, a small retry budget (for example at most 10 percent extra load), and backoff, otherwise a slow backend plus retries becomes a retry storm that overloads the rest. See Timeouts, Retries & Backoff and Fault Tolerance & Circuit Breakers.
Three more failure modes are worth naming. Slow-but-alive backends are worse than dead ones: they pass health checks, hold connections, and attract traffic under least-connections only if they are fast, so watch latency-based ejection. Cascading overload: when one backend is ejected, its share moves to the others, which may then fail too; many balancers have a "panic" threshold (Envoy's panic mode) where, if too large a fraction of hosts are unhealthy, they route to all hosts instead of piling load on the few "healthy" ones. The balancer itself is a single point of failure unless it is redundant (multiple instances behind DNS or anycast, or a managed cloud LB that is redundant internally); state such as sticky tables or connection tracking is what makes failover harder.
Classic web tier LLM inference tier
---------------- ---------------------------------------------
request cost: ~ms, similar request cost: 0.2 s to minutes; varies ~100x with output length
one request = one thread briefly one request holds a slot + KV-cache memory for its whole decode
"in-flight count" ~ load in-flight count is a weak proxy: 2 long requests can outweigh 20 short
new replica ready in seconds new GPU replica: minutes (pull image, load weights, warm up)
stateless partial state: KV cache / prefix cache on the replica's GPU
Better routing signals (system-specific; check your serving stack):
- queue depth: running + waiting requests reported by the server (vLLM exposes such metrics on /metrics)
- outstanding tokens: sum of (prompt tokens + expected output tokens) per replica
- KV-cache utilisation: a replica near its memory limit will queue or preempt
- prefix affinity: send requests sharing a long prompt prefix to the replica that already cached itThe underlying issue is that inference cost is proportional to tokens, not requests, and output length is not known up front (you only know max_tokens, an upper bound). A replica running continuous batching (see Batching & Dynamic Batching) can serve many sequences at once, so "connections" is not even capped at one; what limits it is KV-cache memory and per-step latency, which both grow with the total tokens in flight. Routing on request count therefore mis-estimates load, and queue-depth or token-aware routing (least outstanding tokens, or the server's own running-plus-waiting figure) tracks the real constraint. The toy simulation below shows the direction of the effect; a production version has to estimate output length (from prompt length, task type, or max_tokens) because it is not known in advance.
Three operational differences follow. First, readiness must mean "weights loaded and a warm-up request served", and new replicas should be ramped in gradually. Second, long streaming responses (see Streaming Responses) hold a connection for many seconds, so connection draining on scale-in or deploy must wait for them or let clients reconnect; killing a replica mid-stream loses the partial answer. Third, prefix-cache affinity pulls against balance: sending requests that share a system prompt to one replica saves prefill compute but can create a hot replica, so schedulers combine an affinity score with a load penalty. Some inference gateways and projects in the Kubernetes ecosystem implement this kind of endpoint picking, but the exact features vary, so check current docs. Capacity follow-through lives in Autoscaling for Inference Workloads and Model Serving Architectures; admission control belongs with Rate Limiting & Throttling.
Three small, seeded simulations (standard library only). The first compares round robin, random, least connections and power of two choices on the same heavy-tailed workload: 8 single-worker FIFO servers at about 85 percent utilisation. The second shows how many keys move when a server is added under modulo hashing versus a hash ring. The third is a toy LLM-style comparison of routing by request count versus by outstanding tokens. These are toy models (perfect, instantaneous load information, one balancer, no network), meant to show relative behaviour, not to predict production numbers.
# Python 3: heavy-tailed workload, four routing policies
import random
from collections import deque
N_SERVERS = 8
N_REQ = 20000
LOAD = 0.85 # target utilisation
def make_workload(seed):
rng = random.Random(seed)
# heavy-tailed service time: Pareto(alpha=1.5), scale 0.05s, capped at 20s
svc = [min(0.05 * rng.paretovariate(1.5), 20.0) for _ in range(N_REQ)]
mean_svc = sum(svc) / N_REQ
rate = LOAD * N_SERVERS / mean_svc # arrivals per second
t, arr = 0.0, []
for _ in range(N_REQ):
t += rng.expovariate(rate)
arr.append(t)
return arr, svc, mean_svc
def simulate(policy, arr, svc, seed=1):
rng = random.Random(seed)
finish = [deque() for _ in range(N_SERVERS)] # FIFO single worker: finish times ascending
last_done = [0.0] * N_SERVERS
rr = 0
lat = []
for t, s in zip(arr, svc):
for q in finish: # drop completed requests
while q and q[0] <= t:
q.popleft()
if policy == "round_robin":
i = rr; rr = (rr + 1) % N_SERVERS
elif policy == "random":
i = rng.randrange(N_SERVERS)
elif policy == "least_connections":
i = min(range(N_SERVERS), key=lambda k: len(finish[k]))
elif policy == "power_of_two":
a, b = rng.sample(range(N_SERVERS), 2)
i = a if len(finish[a]) <= len(finish[b]) else b
start = max(t, last_done[i])
last_done[i] = start + s
finish[i].append(last_done[i])
lat.append(last_done[i] - t)
lat.sort()
pick = lambda p: lat[int(p * (len(lat) - 1))]
return sum(lat) / len(lat), pick(0.50), pick(0.99), lat[-1]
arr, svc, mean_svc = make_workload(42)
print(f"servers={N_SERVERS} requests={N_REQ} mean service={mean_svc:.3f}s target utilisation={LOAD}")
print(f"{'policy':<18}{'mean':>8}{'p50':>8}{'p99':>9}{'max':>9}")
for p in ["round_robin", "random", "least_connections", "power_of_two"]:
m, p50, p99, mx = simulate(p, arr, svc)
print(f"{p:<18}{m:8.2f}{p50:8.2f}{p99:9.2f}{mx:9.2f}")
# Real output from python3:
# servers=8 requests=20000 mean service=0.142s target utilisation=0.85
# policy mean p50 p99 max
# round_robin 2.49 0.66 19.15 22.25
# random 3.22 0.95 24.73 30.66
# least_connections 0.44 0.16 3.88 20.66
# power_of_two 0.62 0.26 5.15 20.94
# Python 3: rebalancing cost, modulo hash vs consistent-hash ring
import bisect, hashlib
def h(s):
return int(hashlib.md5(s.encode()).hexdigest(), 16)
def mod_pick(key, servers):
return servers[h(key) % len(servers)]
class Ring:
def __init__(self, servers, vnodes=100):
self.ring = sorted((h(f"{s}#{v}"), s) for s in servers for v in range(vnodes))
self.points = [p for p, _ in self.ring]
def pick(self, key):
i = bisect.bisect(self.points, h(key)) % len(self.ring)
return self.ring[i][1]
keys = [f"user-{i}" for i in range(20000)]
before = [f"s{i}" for i in range(8)]
after = before + ["s8"] # add a 9th server
moved_mod = sum(mod_pick(k, before) != mod_pick(k, after) for k in keys)
r1, r2 = Ring(before), Ring(after)
moved_ring = sum(r1.pick(k) != r2.pick(k) for k in keys)
print(f"add 1 server to 8 -> keys remapped, hash % N : {moved_mod/len(keys):.1%}")
print(f"add 1 server to 8 -> keys remapped, hash ring : {moved_ring/len(keys):.1%} (ideal 1/9 = {1/9:.1%})")
for vn in (1, 100):
r = Ring(before, vnodes=vn)
load = {}
for k in keys:
load[r.pick(k)] = load.get(r.pick(k), 0) + 1
print(f"vnodes={vn:<3} keys per server min/max: {min(load.values())}/{max(load.values())} (even = {len(keys)//8})")
# Real output from python3:
# add 1 server to 8 -> keys remapped, hash % N : 89.0%
# add 1 server to 8 -> keys remapped, hash ring : 10.2% (ideal 1/9 = 11.1%)
# vnodes=1 keys per server min/max: 16/5398 (even = 2500)
# vnodes=100 keys per server min/max: 2344/2709 (even = 2500)
# Python 3: toy LLM pool, route by request count vs by outstanding tokens
import random
from collections import deque
# LLM-style requests: cost ~ output tokens (heavy tailed). 4 GPU replicas, each serves one request at a time (toy).
N=4; REQ=6000
rng=random.Random(7)
tokens=[min(int(40*rng.paretovariate(1.3)),4000) for _ in range(REQ)]
TOK_PER_S=50.0 # illustrative decode speed for the toy model
svc=[t/TOK_PER_S for t in tokens]
rate=0.80*N/(sum(svc)/REQ)
t=0; arr=[]
for _ in range(REQ):
t+=rng.expovariate(rate); arr.append(t)
def run(policy):
fin=[deque() for _ in range(N)]; last=[0.0]*N; lat=[]
for a,s,tok in zip(arr,svc,tokens):
for q in fin:
while q and q[0][0]<=a: q.popleft()
if policy=="least_requests":
i=min(range(N),key=lambda k:len(fin[k]))
else: # least outstanding tokens (uses a size estimate; here exact)
i=min(range(N),key=lambda k:sum(x[1] for x in fin[k]))
st=max(a,last[i]); last[i]=st+s; fin[i].append((last[i],tok)); lat.append(last[i]-a)
lat.sort(); return sum(lat)/len(lat), lat[int(.5*(len(lat)-1))], lat[int(.99*(len(lat)-1))]
print(f"replicas={N} requests={REQ} mean tokens={sum(tokens)/REQ:.0f} max={max(tokens)}")
for p in ("least_requests","least_tokens"):
m,p50,p99=run(p); print(f"{p:<15} mean={m:6.2f}s p50={p50:6.2f}s p99={p99:7.2f}s")
# Real output from python3:
# replicas=4 requests=6000 mean tokens=136 max=4000
# least_requests mean= 6.93s p50= 2.82s p99= 68.67s
# least_tokens mean= 5.99s p50= 2.98s p99= 41.41sRead the first table carefully. The maximum is similar for every policy because a single 20-second request cannot be made faster by any balancer; what routing changes is how much other traffic waits behind it. Least connections beats power of two choices here because the simulation gives it perfect, instant global state on a single balancer; the argument for P2C is that it stays close to that with only two probes and no shared state. In the third simulation, the "least tokens" policy uses the true token count, which a real balancer only has as an estimate, so treat the gain as an upper bound.
Q: Round robin vs least connections: when does each win?
A: Round robin wins on simplicity when requests are short and similar, because equal counts then mean equal work and it needs no state. Least connections wins when durations vary a lot, since in-flight count approximates queue length and steers new work away from a backend stuck on slow requests. In the seeded demo above (heavy-tailed durations, 85 percent utilisation) least connections cut p99 from about 19 s to about 4 s compared with round robin; the exact numbers are toy, the direction is the point. Least connections costs you a counter per backend and, with several balancers, stale state.
Q: What is power of two choices and why use it?
A: You pick two backends at random and send the request to the one with fewer in-flight requests. It needs only two lookups and no global view, so it scales to many balancers, yet a small amount of choice removes most of the imbalance of pure random assignment (max load drops from about log n / log log n to about log log n in the classic analysis). It avoids the herd behaviour of global least-connections when counts are stale, at the price of being slightly worse than perfect least-connections when state is accurate.
Q: Why is modulo hashing a bad way to pick a cache node?
A: Because hash(key) % N changes for almost every key when N changes; in the demo, going from 8 to 9 servers remapped 89 percent of keys. For a cache that is a mass miss, so the database takes the full load exactly when you were adding capacity or lost a node. Consistent hashing with virtual nodes moves about 1/(N+1) of keys instead (10.2 percent in the demo), at the cost of a lookup structure and hot-key skew.
Q: Layer 4 or layer 7 for a gRPC service, and why?
A: Generally L7 or client-side load balancing, because gRPC multiplexes many calls over a long-lived HTTP/2 connection, and an L4 balancer picks a backend per connection, so one client can saturate one backend while others idle. An L7 proxy that understands HTTP/2 balances per request. The costs are extra CPU and latency per request and TLS termination at the proxy; L4 is still fine in front of the L7 tier for raw availability.
Q: How do health checks fail you?
A: A shallow check (process is up) passes while the service cannot actually work, for example a GPU replica whose model failed to load or whose CUDA context died, so traffic keeps flowing to it. A deep check that calls a dependency can flap or take the whole pool out when the dependency blips. Aggressive thresholds cause flapping; lenient ones extend the error window (3 failures at 5 s is up to ~15 s of errors, illustrative). The usual answer is a cheap readiness check plus passive outlier ejection on real errors and latency, and a panic threshold so a mass failure does not overload the survivors.
Q: Are sticky sessions a good idea?
A: Usually a compromise. They let backends keep per-user state in memory, but they skew load, lose sessions when a backend dies or scales in, and complicate deploys. I would make backends stateless with state in Redis or a database and keep affinity only where it is a pure optimisation, such as cache or KV-prefix locality, where losing the affinity costs speed and not correctness.
Q: How would you load balance an LLM inference fleet?
A: Not by request count, since output length varies by orders of magnitude and each request holds GPU memory for its whole decode. I would route on the serving engine's own signals (running plus waiting requests, KV-cache utilisation) or on estimated outstanding tokens, add a prefix-cache affinity term with a load cap, gate readiness on model load plus warm-up, and drain streams on scale-in. The trade-off is that an output-length estimate can be wrong and per-replica metrics are slightly stale, so combine routing with admission control and timeouts.
Follow-up: Your least-connections balancer has 6 instances and one backend gets hammered right after it recovers. Why, and what do you do?
A: A freshly recovered backend has zero in-flight requests, so every balancer sees it as the least loaded and all send it traffic at once (a herd, also cold caches and JIT or model warm-up make it slow). Fixes are slow start (ramp its weight over 30 to 60 seconds, illustrative), power of two choices instead of global minimum so the herd is diluted, and a concurrency cap or queue limit on the backend so it sheds rather than collapses.
Follow-up: How would you handle one very hot key when routing by consistent hash?
A: Hashing preserves skew, so a hot key overloads its backend. Options are to replicate hot keys to several backends and pick among them (detect hot keys by counting), to add a small in-process or edge cache in front, to use consistent hashing with bounded loads so a backend spills excess to its neighbour, or to salt the key (key#0..k) and merge results if the operation allows it. Each trades cache efficiency or consistency for spreading the load.
Follow-up: A backend is slow but still passes health checks. How does the balancer cope?
A: Least-connections and P2C partly self-correct because a slow backend accumulates in-flight requests and receives fewer new ones, whereas round robin keeps feeding it. Beyond that, use passive outlier detection on latency or 5xx rate to eject or de-weight it, hedge or retry on another backend with a budget, and set per-request timeouts so slow work fails fast. The trade-off is that latency-based ejection can misfire when all backends slow down together, so cap the fraction ejected.
In a typical design the request path is DNS or anycast to an edge L4 balancer, then an L7 tier (gateway or ingress) that terminates TLS, authenticates, rate limits, and routes by path or model, then service-level balancing (sidecars, client-side, or a mesh) to stateless replicas. As an estimation, assume 20,000 requests per second at peak and a backend that can handle about 500 per second at a healthy 70 percent utilisation: you need 20,000 / 350 = roughly 57 instances, plus enough headroom to lose one availability zone (with three zones, provision for 1.5x, so around 86), which is why "N+1" and zone-aware balancing come up. For an LLM tier, do not guess a concurrency-per-replica figure; size by tokens: if a replica sustains T output tokens per second in aggregate (measure it under your batch settings) and peak demand is R requests per second at L average output tokens, you need roughly R x L / T replicas plus headroom, and you route by queue depth so the tail of long generations does not starve short ones. Mention the failure story explicitly: health checks with sensible thresholds, connection draining for streams, retries with a budget, a panic threshold, and redundancy of the balancer itself. If a cache tier is involved, name consistent hashing and what happens to hit rate when a node dies.
Load balancing trades a little latency, a new component and some state for horizontal scale and availability. Choosing the algorithm is a choice about which signal you trust: round robin trusts nothing and is cheap but ignores work; least connections trusts a fresh global counter and can herd; P2C trusts two random samples and is close to optimal at a fraction of the cost; hashing trusts a key and gives you locality at the cost of hotspots and rebalancing pain; token-aware routing trusts an estimate that can be wrong. L7 gives smarter routing and observability but costs CPU, latency and a bigger blast radius if the proxy misbehaves. Stickiness improves cache hit rates and simplifies stateful code but makes failures and scale-in messier. The whole approach is the wrong tool when the bottleneck is downstream (a shared database or a rate-limited provider API: more balanced replicas only push more load into the same choke point), when a single request is so large that only vertical scaling or sharding the work helps, or when a queue-based design (see Synchronous vs Asynchronous Processing) would absorb bursts better than spreading them.