Autoscaling for inference means adding and removing model-serving replicas (usually GPU-backed) automatically as demand changes. It looks like a solved web-tier problem and is not: GPU replicas take minutes to become useful, the usual CPU-utilization signal barely correlates with LLM load, and idle GPUs are expensive. Interviewers probe it because the honest answer is a set of trade-offs between latency, cost and stability, and because it exposes whether you understand queueing rather than just Kubernetes YAML. For a Backend or AI engineer the questions are: which metric do you scale on, how do you survive a burst that arrives faster than replicas can start, and how do you stop the autoscaler from flapping or bankrupting you.
Autoscaling is automatically turning capacity up or down to match demand, without a human deciding in the moment. For an AI model serving predictions, that means starting more copies of the model when lots of people are asking it questions, and shutting copies down when traffic quiets, so the service stays fast without paying for idle machines around the clock.
Think of a restaurant that calls in extra cooks only when a dinner rush hits, and sends them home once the crowd thins out. The kitchen doesn't keep a full staff on the clock all day just in case — it watches how many orders are coming in and adjusts staffing to match, so food still comes out quickly without paying cooks to stand around during slow hours.
Signal Good for Weakness
------------------------- ---------------------------- ------------------------------------------
CPU utilization stateless web / CPU models GPU work barely moves CPU; poor for LLMs
GPU utilization rough busy/idle "busy" does not mean "saturated"; a GPU can
show high util with an empty queue, or
look fine while requests queue
Request rate (req/s) simple, predictable requests differ wildly in cost (tokens)
In-flight concurrency matches Little's law needs per-replica concurrency limit
Queue depth / wait time direct sign of overload lagging; reacts after users wait
Tokens/s (in and out) LLM capacity in real units need a measured per-replica capacity
KV-cache usage engine-specific saturation metric names/semantics differ by engine
Little's law (steady state): in-flight = arrival rate x average latency
ILLUSTRATIVE: 150 req/s x 4 s = 600 in flight; if one replica holds ~32 concurrent
sequences, replicas = ceil(600 / 32) = 19 (plus headroom, e.g. target 70% -> ~27)A good scaling signal moves before or as users are hurt, is proportional to load, and is cheap to measure. For LLM serving the expensive resource is GPU memory bandwidth and KV-cache space, and the number of sequences a replica can hold concurrently is bounded by the KV cache (see LLM Inference Fundamentals), so in-flight concurrency and queue depth track saturation far better than CPU. Utilization from tools such as NVIDIA DCGM is a coarse "was any kernel running" figure and can stay high while latency is fine or low while a queue builds, which is why it is generally a secondary signal (see GPU Hardware Utilization Basics). Engines such as vLLM expose Prometheus metrics for running and waiting requests and cache usage, but the exact metric names vary by version, so check the docs of the engine you run.
Requests are an imperfect unit because a 4,000-token prompt costs far more than a 20-token one, so many teams scale on tokens per second or on concurrency against a measured per-replica capacity, then add queue wait as a guard. This connects directly to Capacity Planning & SLOs: you measure what one replica sustains at your latency SLO (load-test it, do not guess), and the autoscaler's job is to keep utilization near a target below 100% because queueing delay explodes as utilization approaches 1.
Timeline of a reactive scale-up (ILLUSTRATIVE durations):
t=0 burst begins, replicas saturated, queue grows
t=10 metric scraped / aggregated (scrape interval + HPA sync period)
t=15 autoscaler decides "need +N"
t=15-60 GPU node provisioned (if cluster autoscaler must add a node)
t=60-90 pull image, load weights into GPU, warm up, pass readiness probe
t=90 new replica finally takes traffic
-> users waited ~90 s behind a queue that grew the whole time
Queue growth during the lag = (offered rate - current capacity) x lag
e.g. (150 - 40 req/s) x 90 s = 9,900 queued requests -> more than a minute of backlog
even after the new replicas arrive (they must ALSO drain the backlog)Reactive autoscaling acts on what already happened, so its damage is set by the provisioning lag, not by how clever the policy is. If a burst is shorter than the lag, reactive scaling never helps at all: replicas arrive after the burst ends and you pay for them anyway. The remedies all buy time: keep headroom (a warm pool or higher minimum replicas), scale on a leading signal such as queue growth rate rather than a lagging average, use predictive or scheduled scaling when load has a pattern (a daily peak, a known launch, a batch job at 09:00), pre-pull images and cache weights on local disk to shorten the lag, and shed or queue with a bounded wait so overload degrades gracefully instead of unboundedly. Predictive scaling depends on the pattern being real; a forecast that is wrong scales up for nothing or misses the spike, so it is usually combined with a reactive floor rather than trusted alone.
Scale-to-zero arithmetic (ILLUSTRATIVE; assume $2 per GPU-hour, check real pricing):
1 GPU idle 20 h/day = $40/day wasted (about $1,200/month)
saved by scale-to-zero, but first request after idle waits the full cold start
(node + image + weights) = tens of seconds to minutes
Flapping (no stabilization):
load: high low high low high
reps: 20 5 20 5 20 <- each scale-up pays the 90 s lag AND the GPU cost while starting
fix: scale UP fast, scale DOWN slowly (stabilization window / cooldown),
and scale down on a smoothed signal, not a single low sample
Kubernetes HPA core rule: desiredReplicas = ceil( currentReplicas x currentMetric / targetMetric )
(HPA also has a scale-down stabilization window; the default is generally 300 s,
scale-up has none by default - check your Kubernetes version's docs)Scale-to-zero is attractive for rarely used models and dev environments because a GPU that is idle most of the day is pure waste, but the price is a cold start on the first request after idle, which for a large model can be far above any interactive SLO. Users of scale-to-zero either accept the wait (batch and internal tools), show a "warming up" experience, or keep a small always-on floor for anything user-facing. A warm pool is the middle path: hold N replicas beyond current need so a burst of up to N replicas' worth of extra load is absorbed immediately while the autoscaler orders more. Its cost is a fixed idle-GPU bill, which you compare against the cost of SLO violations.
Flapping happens when the autoscaler oscillates because scale-down reacts to noise. Because scale-up is slow and expensive for GPUs, aggressive scale-down is the wrong default: killing a replica means losing its loaded model, its warm KV and prefix caches, and any in-flight requests, unless you drain gracefully (stop routing, let running generations finish, then terminate; see Fault Tolerance & Circuit Breakers and Load Balancing for the drain and health-check side). The demo below shows the trade in numbers: a cooldown cuts scale events but keeps more replicas alive afterwards.
# Sketch only, not run here. Field names follow the documented autoscaling/v2 and KEDA APIs;
# verify against the versions you deploy.
# HPA on a custom per-pod metric (needs a metrics adapter such as Prometheus Adapter):
# apiVersion: autoscaling/v2
# kind: HorizontalPodAutoscaler
# spec:
# scaleTargetRef: { kind: Deployment, name: llm-server }
# minReplicas: 3
# maxReplicas: 30
# metrics:
# - type: Pods
# pods: { metric: { name: inflight_requests }, target: { type: AverageValue, averageValue: "24" } }
# behavior:
# scaleDown: { stabilizationWindowSeconds: 600 }
# KEDA ScaledObject scaling on a Prometheus query (e.g. queue depth), can allow minReplicaCount: 0
# triggers:
# - type: prometheus
# metadata: { query: "sum(queue_depth)", threshold: "20" }
Layers that all need to scale: pods (HPA/KEDA) -> nodes (Cluster Autoscaler / Karpenter) -> GPU quota
a pending GPU pod only starts after a GPU node exists; node provisioning is often the slowest step.At a working level: the Horizontal Pod Autoscaler compares an observed metric with a target and adjusts replicas, and it can use resource metrics, custom per-pod metrics or external metrics; KEDA builds on it, adds event-driven triggers (queue length, Prometheus queries, message backlogs), and supports scaling to zero. Neither creates GPU nodes; a node-level autoscaler (Cluster Autoscaler or Karpenter, depending on the cluster) must add capacity when pods are pending, which adds node boot and driver time, and cloud GPU quotas or availability can cap you regardless of policy. Rolling out new model versions while autoscaling adds another interaction covered in A/B Testing & Canary Deployments. See Container Orchestration & Kubernetes Basics for the object model and Batching & Dynamic Batching for why per-replica capacity depends on how batching is tuned.
import random, math
random.seed(11)
T = 1000 # simulated seconds
CAP = 10 # requests/s one replica can serve (ILLUSTRATIVE)
SLO_S = 2.0 # latency SLO: queue wait + service <= 2 s
PROVISION_S = 90 # GPU node + image pull + model load before a replica takes traffic
TARGET = 7 # target req/s per replica (70% utilisation)
def rate(t): # bursty offered load, req/s
if 300 <= t < 420: return 150
if 700 <= t < 760: return 100
return 30
def poisson(lam): # Knuth; fine for small lam, normal approx for large
if lam > 30:
return max(0, int(random.gauss(lam, math.sqrt(lam)) + 0.5))
L, k, p = math.exp(-lam), 0, 1.0
while True:
p *= random.random()
if p <= L: return k
k += 1
ARR = [poisson(rate(t)) for t in range(T)] # same arrivals for every policy
def run(name, min_rep, cooldown, scheduled=False, initial=None):
ready = initial or min_rep
pending = [] # (ready_at, count)
queue, viol, total, max_q = 0, 0, 0, 0
replica_s, events, last_down, desired_hist = 0, 0, -10**9, []
for t in range(T):
# replicas finishing provisioning join now
for r in [p for p in pending if p[0] <= t]:
ready += r[1]; pending.remove(r)
# --- autoscaler decision every 10 s, from the last 10 s of arrivals ---
if t % 10 == 0 and t >= 10:
observed = sum(ARR[t-10:t]) / 10
want = math.ceil((observed + queue / 30) / TARGET) # rate + backlog (drain in ~30 s)
if scheduled: # predictive: know the calendar, act early
for start, end, r in [(300, 420, 150), (700, 760, 100)]:
if start - PROVISION_S - 10 <= t < end:
want = max(want, math.ceil(r / TARGET))
want = max(want, min_rep)
have = ready + sum(c for _, c in pending)
if want > have:
pending.append((t + PROVISION_S, want - have)); events += 1
elif want < ready and t - last_down >= cooldown:
ready = max(want, min_rep); events += 1; last_down = t
# --- serve this second ---
a = ARR[t]
cap_now = ready * CAP
wait = queue / cap_now if cap_now else 99
lat = wait + 1.0 / CAP
total += a
if lat > SLO_S: viol += a
queue += a
queue -= min(queue, cap_now)
max_q = max(max_q, queue)
replica_s += ready + sum(c for _, c in pending) # you pay while provisioning too
print(f"{name:34s} SLO-miss {100*viol/total:5.1f}% max queue {max_q:5d} replica-s {replica_s:6d} scale events {events:3d}")
print(f"requests total = {sum(ARR)}, peak offered = 150 req/s, one replica = {CAP} req/s, provision = {PROVISION_S}s")
run("reactive, min=4, no cooldown", 4, 0)
run("reactive, min=4, cooldown 120s", 4, 120)
run("reactive + warm pool min=9", 9, 120)
run("reactive + warm pool min=12", 12, 120)
run("scheduled pre-scale, min=4", 4, 120, scheduled=True)
run("static peak (15 replicas always)", 15, 10**9)
# Real output from python3 (seeded; a toy fluid-queue model with ILLUSTRATIVE parameters, not a benchmark):
# requests total = 48362, peak offered = 150 req/s, one replica = 10 req/s, provision = 90s
# reactive, min=4, no cooldown SLO-miss 55.9% max queue 10973 replica-s 14770 scale events 39
# reactive, min=4, cooldown 120s SLO-miss 53.9% max queue 10973 replica-s 21590 scale events 25
# reactive + warm pool min=9 SLO-miss 46.5% max queue 5973 replica-s 15530 scale events 17
# reactive + warm pool min=12 SLO-miss 35.5% max queue 2973 replica-s 15600 scale events 14
# scheduled pre-scale, min=4 SLO-miss 0.0% max queue 5 replica-s 12280 scale events 13
# static peak (15 replicas always) SLO-miss 0.0% max queue 121 replica-s 20500 scale events 3
# Reading it: the 60 s second burst is shorter than the 90 s provisioning lag, so reactive policies
# never help with it; the scheduled policy pre-scales before both bursts and meets the SLO at ~60%
# of the static-peak replica-seconds; the cooldown cuts scale events but keeps replicas (cost) longer;
# a warm pool reduces but does not remove the miss unless it is nearly peak-sized.Q: Why is CPU a poor autoscaling signal for LLM serving?
A: The bottleneck is GPU memory bandwidth and KV-cache capacity, not host CPU, so CPU stays low while the GPU-bound queue grows. Better signals are in-flight concurrency, queue depth or wait time, and tokens per second against a measured per-replica capacity. GPU utilization helps but is coarse: it says a kernel was running, not that the replica is saturated.
Q: How would you pick the target concurrency or utilization for a replica?
A: Load-test one replica at increasing concurrency until latency crosses the SLO, note the knee, and target roughly 60-80% of that point so bursts do not immediately violate it. Queueing theory says wait time rises steeply near saturation, so running at 95% leaves no slack. The trade-off is idle capacity: a lower target costs more GPUs but buys latency stability.
Q: Why does reactive scaling fail for bursty traffic?
A: It cannot react faster than the metric interval plus the provisioning lag, which for GPUs is often a minute or more once node start, image pull and weight load are counted. During the lag the queue grows at (offered minus capacity), and afterwards the new capacity must also drain that backlog. If the burst is shorter than the lag, the replicas arrive too late to help, which my simulation shows with a 60 s burst against a 90 s lag.
Q: Warm pool, scheduled scaling, or scale-to-zero: how do you choose?
A: By how predictable and latency-sensitive the traffic is. Known daily or launch peaks suit scheduled or predictive scaling, unpredictable bursts on interactive endpoints need a warm pool or higher min replicas (paying idle GPU-hours for latency), and rarely used or internal models can scale to zero and accept the cold start. Many systems mix them: an always-on floor, scheduled pre-scaling for the predictable part, reactive scaling for the rest.
Q: What is flapping and how do you prevent it?
A: The autoscaler repeatedly scales up and down because it reacts to noisy or short-lived changes, and with GPUs each up-scale pays a long start and each down-scale throws away warm state. Prevent it with asymmetric policy (fast up, slow down), a stabilization window or cooldown on scale-down, a smoothed metric, and a hysteresis gap between the scale-up and scale-down thresholds. The cost is that you hold extra replicas longer after a spike.
Q: How does Kubernetes HPA decide the replica count, and what does it not do?
A: It computes roughly ceil(current replicas x current metric / target metric), bounded by min and max, with configurable behaviour such as stabilization windows. It only scales pods: it does not add GPU nodes (a node autoscaler must), it does not know your model-load time, and its metric pipeline adds delay. KEDA extends this with event-driven triggers and scale-to-zero.
Q: How do you protect the system while capacity is still being provisioned?
A: Bound the queue and shed or degrade rather than let latency grow without limit: reject with 429 and Retry-After beyond a wait threshold, prioritize interactive over batch traffic, route overflow to a smaller model or another provider, and apply per-tenant limits (see Rate Limiting & Throttling). This keeps the admitted requests within SLO at the cost of visible rejections.
Follow-up: Your autoscaler scaled up but p99 stayed bad for several more minutes. Why?
A: New replicas must also drain the backlog built during the lag, and they start with cold caches, so they are slower than steady-state replicas. If the metric included only rate and ignored queue depth, the autoscaler may also have scaled down too early. Check replica readiness time, queue depth over time, and whether the load balancer sent traffic before warm-up finished.
Follow-up: How would you scale a service with three different models on the same cluster?
A: Give each model pool its own metric, min/max and target since their capacities and load differ, and treat GPU quota and node capacity as the shared resource that limits the total. Priorities decide who loses when quota is exhausted; a shared warm pool of generic nodes can cut node-provisioning lag. See Multi-Model & Multi-Tenant Serving for sharing GPUs across models.
Follow-up: Would you autoscale on tokens per second or on queue depth?
A: Tokens per second (or concurrency against measured capacity) is a proportional signal that lets you compute how many replicas you need before overload, while queue depth is a direct but lagging overload indicator. I would use a proportional signal as the primary and queue wait as a safety trigger. The weakness of tokens per second is that you must know per-replica capacity for your prompt and output length mix, which drifts as traffic changes.
In an LLM-serving or model-API design, put autoscaling next to capacity planning: state the ASSUMED demand (for example 100 req/s peak, 500 output tokens per response, so about 50,000 output tokens/s at peak), the measured per-replica capacity (say 2,500 output tokens/s per replica after load testing, ILLUSTRATIVE), and the arithmetic that follows: 50,000 / 2,500 = 20 replicas at 100%, about 29 at a 70% target. Then explain how you get there in time: a floor of replicas for baseline, scheduled scaling for the known daily peak, reactive scaling on concurrency and queue wait for the rest, and load shedding while capacity boots. Cover the layers (pods, then nodes, then GPU quota), the provisioning lag as a number you have measured, the warm-pool cost in GPU-hours against the cost of missed SLOs, and the graceful drain on scale-down. Tie it to Capacity Planning & SLOs, Cost Optimization for AI Systems and the gateway-level controls in Design an LLM API Gateway, and mention multi-region failover as the answer when one region's GPU supply is the limit.
Autoscaling trades cost against latency and stability. Aggressive scaling tracks demand closely and saves GPU-hours but exposes users to the provisioning lag and risks flapping; conservative scaling with warm pools and slow scale-down keeps latency steady but pays for idle GPUs; scheduled scaling is cheap and effective for predictable load but useless for surprises. Scale-to-zero is the cheapest when idle and the worst for the first request. The extra machinery (custom metrics, KEDA, node autoscaling, drain logic) is itself a source of bugs and lag. Autoscaling is the wrong tool when load is flat and predictable (provision statically and save the complexity), when the model is so large that startup takes tens of minutes (scaling cannot keep up; use fixed capacity plus admission control), or when GPU supply, not policy, is the limit; there, capacity reservations and load shedding matter more than a cleverer autoscaler.