Capacity planning answers "how much infrastructure do we need so that users get the promised experience", and an SLO (service level objective) is the promise that turns that vague question into arithmetic: for example "99.9% of requests succeed and 99% of first tokens arrive within 800 ms, measured over 30 days". Interviewers probe it because it separates engineers who can size a system from ones who can only draw boxes: can you go from "2M requests a day" to QPS, concurrency, GPU count and a monthly bill, and say what happens when you run the fleet too hot? For an AI Backend engineer it is sharper than for ordinary web services, because GPUs are expensive, scale up slowly (see Kubernetes basics) and latency depends on batching. Every number below is ILLUSTRATIVE, built from stated assumptions; real sizing comes from your own load tests and traffic data.
Capacity planning is the work of figuring out, ahead of time, how much computing power (servers, storage, network) a system needs so it can handle its expected traffic without slowing down or falling over. It means translating a business number, like "we expect a million users next month", into a concrete shopping list of machines, and doing that early enough to buy or provision them before they're actually needed.
An SLO (service level objective) is a specific, measurable promise about how well a service should perform, such as "99.9% of requests succeed" or "pages load in under one second, 99% of the time". It turns a vague goal like "be reliable" into a number a team can actually track, aim for, and be held accountable to.
Think of a restaurant getting ready for a big holiday weekend. Capacity planning is deciding how many chefs, tables, and how much food to have on hand based on how many guests are expected. An SLO is the promise the restaurant makes to its guests, like "your table will be ready within 15 minutes of your reservation, 95% of the time." If the restaurant under-plans, the SLO gets missed and guests wait in the cold; if it over-plans, it wastes money on staff and food that go unused.
SLI (indicator) a measurement: good_events / valid_events
e.g. requests answered with status < 500 AND first token < 800 ms
SLO (objective) a target for the SLI: "99.9% over a rolling 30 days"
SLA (agreement) a contract with consequences (credits, penalties); usually looser than the internal SLO
Error budget 1 - SLO = 0.1% of events may be bad in the window
30 days = 43,200 min -> 99.9% allows 43.2 min of full outage
availability 99% -> 432 min (7.2 h) 99.9% -> 43.2 min
availability 99.99% -> 4.32 min 99.999% -> 0.43 min
Burn rate = observed_error_rate / (1 - SLO) 1.0 = spending the budget exactly over the window
14.4x burn for 1 h = 2% of a 30-day budget (a common "page someone" threshold)
6x burn for 6 h = 5% (a common ticket / slower alert)An SLI must describe something the user experiences, measured as close to the user as practical: end-to-end success and latency, not GPU utilisation. For LLM products that usually means a success rate, time to first token (TTFT), inter-token latency or total time, and sometimes a quality proxy. The SLO is deliberately below 100%, because 100% is unattainable and the last 9 costs disproportionately. What you get in exchange is an error budget: a pool of allowed failure you can spend on risky launches, migrations, and experiments; when it is exhausted, the policy is to slow feature work and fix reliability. That converts the "ship faster versus be reliable" argument into a number both sides agreed on beforehand. Burn-rate alerting (alerting on how fast the budget is being spent, over multiple windows) fires early on severe incidents and stays quiet on slow, harmless drift; the multi-window recipe is described in Google's SRE Workbook, and the demo below computes the same figures. See observability for AI systems for how the SLIs are actually measured and alerted on.
Where it breaks: an SLO on the average hides pain, so use percentiles (p50, p95, p99) on latency and remember you cannot average percentiles across hosts, you must aggregate histograms. Request-based SLOs also treat a 30-second outage at peak like one at 3 a.m. only if traffic is the same, and a time-based one does not; pick which matches your users. Finally, an SLO on a component that users do not perceive ("GPU util above 70%") is a cost metric, not an SLO.
ASSUMPTIONS (all illustrative; state them out loud in an interview)
requests per day 2,000,000
peak / average QPS 4x (traffic is diurnal and bursty)
avg request latency 6 s (streamed chat completion)
tokens per request 1500 in, 400 out
GPU decode throughput 2,000 output tokens/s (ASSUMED aggregate under batching, must be measured)
planned peak utilisation 60%
average QPS = 2,000,000 / 86,400 = 23.1
peak QPS = 23.1 x 4 = 92.6
Little's law L = lambda x W = 92.6 x 6 s = 556 requests in flight
output tok/s = 92.6 x 400 = 37,037
GPUs = 37,037 / (2,000 x 0.6) = 30.9 -> 31, plus 1 spare (N+1) = 32
logs = 2M x (1900 tokens x ~4 B) = ~15 GB/day (~0.46 TB / 30 days)Little's law (L = lambda x W: average items in the system equals arrival rate times average time in the system) is the workhorse: it holds for any stable system regardless of distribution, so it converts QPS and latency into concurrency, which is what connection pools, thread counts, KV-cache memory and GPU batch slots are actually sized against. For LLM serving there are two separate ceilings and you need both: throughput (output tokens per second the GPUs can generate, generally bound by memory bandwidth in decode) and concurrency (how many sequences fit in GPU memory alongside the weights, since each in-flight sequence holds KV cache proportional to its length; see batching and dynamic batching and GPU hardware utilization basics). Take the larger of the two GPU counts. Two things move the numbers a lot: the input/output token mix (long prompts stress prefill compute, long outputs stress decode) and batching efficiency, so "tokens/s per GPU" from a benchmark of a different prompt mix is almost useless; measure with your own distribution.
Storage needs the same discipline: raw text is small, but embeddings, vector indexes, traces with full prompts and retained logs multiply it, and retention policy (plus privacy rules) is usually the lever. Rather than quoting prices, keep cost as a formula: GPU count x hourly rate x hours, and say "check current pricing" for the rate.
M/M/1 queue (one server, random arrivals, exponential service), utilisation rho = lambda / mu:
mean time in system W = S / (1 - rho) S = service time
rho = 0.50 -> 2x S rho = 0.90 -> 10x S
rho = 0.70 -> 3.3x S rho = 0.95 -> 20x S
rho = 0.80 -> 5x S rho = 0.99 -> 100x S
Latency vs utilisation is a hockey stick (shape only, not to scale):
latency
| |
| /
| |
| _/
|________________________________---
+---------|---------|---------|--------|----> utilisation
50% 70% 90% 100%The intuition is that a server with no spare capacity cannot absorb random clustering of arrivals: a burst creates a queue, and at high utilisation the queue takes very long to drain. So average utilisation of 90% does not mean "10% headroom", it means waits an order of magnitude longer than at 50%, and the tail (p99) degrades much earlier and harder than the mean. That is why fleets are planned to run at 50 to 70% at peak, not 95%. M/M/1 is a simplification: real LLM servers batch requests, service times are heavy-tailed (a 4000-token answer is not like a 40-token one), and a pool of servers (M/M/c) queues less than the same capacity split into isolated single servers, which is an argument for shared pools and against over-partitioning per tenant (see multi-model and multi-tenant serving). Continuous batching softens the cliff by admitting new sequences between decode steps, but once the batch and KV cache are full, new requests still queue.
Practical consequences: put a bounded queue and load shedding in front of the servers (fail fast with 429 or 503 rather than let latency explode for everyone, see rate limiting and throttling), scale on queue depth or in-flight requests before utilisation reaches the knee, and set timeouts and retry budgets carefully so retries do not add load to an already saturated system (see timeouts, retries and backoff).
Fleet sizing checklist (per region, per model)
base = peak demand / (per-GPU capacity x target utilisation)
+ failure N+1 (lose one node), or N+2 for big fleets; for zones: survive one zone loss
3 zones, need capacity C after losing 1 -> each zone carries C/2 -> total 1.5C (50% over-provision)
2 zones, lose 1 -> each zone carries C -> total 2C (100% over-provision)
+ growth forecast x safety factor, reviewed each quarter
+ deploy surge/rolling-update slack (a GPU rollout may temporarily lose or need one replica)
+ spike cold start of a new replica is minutes, so headroom or a warm pool, not just autoscaling
Load test plan
1. Use production-like prompt/output length distributions, not a single fixed prompt
2. Ramp arrival rate (open loop) until the SLO breaks; record the knee, not just max throughput
3. Soak test for memory leaks, KV-cache fragmentation, and thermal/limit effects
4. Test failure: kill a replica / a zone and measure recovery time and error burst
5. Test the cold path: scale from minimum to peak and time each stageHeadroom is insurance, and its price is idle GPUs. The zone arithmetic shows the trade clearly: surviving a zone loss with three zones costs 50% extra capacity, with two zones 100%. The alternative is to accept a degraded mode, shedding low-priority traffic, serving a smaller or quantised model, or falling back to another provider, which is cheaper but must be designed and tested (see LLM gateway and multi-provider routing and fault tolerance and circuit breakers). Load testing must be open-loop (requests arrive on a schedule regardless of how slow the server is) because a closed-loop test where each virtual user waits for a reply slows itself down and hides the collapse, a well-known measurement pitfall called coordinated omission.
Cost versus reliability is a curve with a steep end: each extra nine roughly needs redundancy at another level (more replicas, zones, regions, providers, and operational rigour), while the incremental user value falls quickly. A sensible framing: set the SLO from what users tolerate, compute the capacity it implies, then check cost with the levers in cost optimization for AI systems (batching, quantisation, caching, smaller models, reserved versus on-demand versus spot capacity, autoscaling down off-peak). A tier design also helps: interactive traffic gets a tight SLO and reserved capacity, batch and offline traffic gets a loose one and runs on spare or preemptible GPUs.
Pure arithmetic in standard-library Python: Little's law and GPU fleet sizing from the assumptions above, an M/M/1 latency table, and error-budget burn rates. Nothing here is a benchmark; the assumed GPU throughput and memory numbers are placeholders you would replace with measurements.
import math
# ---- ILLUSTRATIVE assumptions (not measurements) ----
requests_per_day = 2_000_000
peak_to_avg = 4 # peak-hour QPS is 4x the daily average
latency_s = 6.0 # end-to-end seconds per streamed chat request
in_tokens, out_tokens = 1500, 400
gpu_decode_tps = 2000 # ASSUMED aggregate output tokens/s per GPU under batching
target_util = 0.60 # plan to run at 60% at peak
kv_gib_per_seq = 0.5 # ASSUMED KV cache per in-flight sequence
seqs_per_gpu_mem = 40 # ASSUMED max concurrent sequences a GPU can hold
avg_qps = requests_per_day / 86_400
peak_qps = avg_qps * peak_to_avg
print(f"avg QPS = {avg_qps:.1f}")
print(f"peak QPS = {peak_qps:.1f}")
# Little's law: L = lambda * W (in-flight requests = arrival rate * time in system)
concurrency = peak_qps * latency_s
print(f"in-flight requests at peak (Little) = {concurrency:.0f}")
# GPU count from throughput
peak_out_tps = peak_qps * out_tokens
gpus_tput = peak_out_tps / (gpu_decode_tps * target_util)
# GPU count from concurrency (memory-bound: how many sequences fit)
gpus_conc = concurrency / (seqs_per_gpu_mem * target_util)
need = max(gpus_tput, gpus_conc)
print(f"peak output tokens/s = {peak_out_tps:,.0f}")
print(f"GPUs by throughput = {gpus_tput:.1f}")
print(f"GPUs by concurrency = {gpus_conc:.1f}")
n = math.ceil(need)
print(f"with N+1 headroom = {n + 1} GPUs (ceil({need:.1f}) + 1 spare)")
# Storage: logging every request/response
bytes_per_req = (in_tokens + out_tokens) * 4 # ~4 bytes of text per token (rough)
gb_day = requests_per_day * bytes_per_req / 1e9
print(f"raw text logs = {gb_day:.1f} GB/day, {gb_day*30/1000:.2f} TB/30 days")
# Queueing: M/M/1 mean time in system W = S / (1 - rho)
print("\nutilisation -> mean latency multiplier (M/M/1, service time S=1)")
for rho in (0.5, 0.7, 0.8, 0.9, 0.95, 0.99):
print(f" rho={rho:.2f} W = {1/(1-rho):6.1f} x S")
# Error budget and burn rate
slo = 0.999
window_days = 30
budget_frac = 1 - slo
minutes = window_days * 24 * 60
print(f"\nSLO {slo:.1%} over {window_days}d -> {budget_frac*minutes:.1f} min of full downtime allowed")
month_requests = requests_per_day * window_days
print(f"error budget = {month_requests * budget_frac:,.0f} failed requests of {month_requests:,}")
def burn(err_rate, hours):
rate = err_rate / budget_frac # burn rate: 1.0 = exactly on budget
used = err_rate * hours / (window_days * 24) # fraction of budget consumed
return rate, used / budget_frac
for err, hrs in ((0.001, 720), (0.0144, 1), (0.05, 0.33)):
r, u = burn(err, hrs)
print(f" error rate {err:.2%} for {hrs:g}h -> burn rate {r:.1f}x, uses {u:.1%} of the 30-day budget"
f"{' (budget gone in %.1f h)' % (window_days*24/r) if r > 1 else ''}")
# Output:
# avg QPS = 23.1
# peak QPS = 92.6
# in-flight requests at peak (Little) = 556
# peak output tokens/s = 37,037
# GPUs by throughput = 30.9
# GPUs by concurrency = 23.1
# with N+1 headroom = 32 GPUs (ceil(30.9) + 1 spare)
# raw text logs = 15.2 GB/day, 0.46 TB/30 days
#
# utilisation -> mean latency multiplier (M/M/1, service time S=1)
# rho=0.50 W = 2.0 x S
# rho=0.70 W = 3.3 x S
# rho=0.80 W = 5.0 x S
# rho=0.90 W = 10.0 x S
# rho=0.95 W = 20.0 x S
# rho=0.99 W = 100.0 x S
#
# SLO 99.9% over 30d -> 43.2 min of full downtime allowed
# error budget = 60,000 failed requests of 60,000,000
# error rate 0.10% for 720h -> burn rate 1.0x, uses 100.0% of the 30-day budget
# error rate 1.44% for 1h -> burn rate 14.4x, uses 2.0% of the 30-day budget (budget gone in 50.0 h)
# error rate 5.00% for 0.33h -> burn rate 50.0x, uses 2.3% of the 30-day budget (budget gone in 14.4 h)Read the output: at peak the system holds about 556 requests in flight, throughput (31 GPUs) is the binding constraint rather than memory (23 GPUs) under these assumed numbers, and one spare GPU brings the fleet to 32. The M/M/1 table shows the same server going from 2x its service time to 10x between 50% and 90% utilisation. The last lines give the standard burn-rate result: a 1.44% error rate is only a 14.4x burn, but holding it for one hour already consumes 2% of the monthly budget, which is why fast-burn alerts are set around there.
Q: What is the difference between an SLI, an SLO and an SLA?
A: An SLI is a measured indicator such as the fraction of requests that succeed within 800 ms, an SLO is the internal target for it (99.9% over 30 days), and an SLA is the external contract with financial or legal consequences, normally set looser than the SLO so you have room before penalties. The trade-off is that a tighter SLO costs more redundancy and slows feature velocity, so it should reflect what users actually need.
Q: What is an error budget and how is it used?
A: It is the allowed amount of failure, 1 minus the SLO: 0.1% of requests, or about 43 minutes of full outage, over 30 days at 99.9%. Teams spend it on releases, experiments and maintenance, and when it is exhausted they freeze risky changes and prioritise reliability work. It replaces subjective arguments between product and operations with an agreed number, but only works if the SLI genuinely reflects user pain.
Q: How would you estimate the number of GPUs for an LLM chat service?
A: Convert requests per day to peak QPS with a peak-to-average factor, multiply by output tokens per request to get required tokens per second, and divide by the measured per-GPU decode throughput times a target utilisation such as 60%. Cross-check memory: peak concurrency from Little's law times KV cache per sequence plus weights must fit, and take the larger answer, then add N+1 or zone-loss headroom. With assumed 2M requests a day, 4x peak, 400 output tokens and 2,000 tokens/s per GPU it comes out around 31 GPUs plus a spare, but those inputs are placeholders until benchmarked.
Q: Explain Little's law and where you have used it.
A: The average number of items in a stable system equals the arrival rate times the average time each spends in it: L = lambda x W. At 90 requests per second and 6 seconds per request there are about 540 requests in flight, which tells you how many connections, worker slots or KV-cache-holding sequences you need. It needs no assumptions about the distribution, but it gives averages, so tails and burstiness still need headroom.
Q: Why not run GPUs at 95% utilisation to save money?
A: Queueing delay grows like 1/(1 - utilisation), so latency near saturation is multiples of the service time, p99 degrades first, and any burst or a lost node pushes the system past capacity, where queues build and timeouts trigger retries that add more load. Running near 60% costs about 1.5x the capacity of running at 90%, and buys tail latency and failure tolerance. If the workload is offline batch with no latency SLO, high utilisation is exactly right.
Q: How do you pick percentile targets for an SLO?
A: Use the percentile that matches who gets hurt: p50 for typical experience, p95 or p99 for the slow tail that users and downstream timeouts notice. For streaming LLMs I set separate targets for time to first token and for inter-token latency or total time, since they depend on different resources (prefill and queueing versus decode). Higher percentiles require more headroom and are noisier to measure, so a p99.9 target needs enough traffic to be statistically meaningful.
Q: How do you load test an LLM service properly?
A: Replay a realistic distribution of prompt and output lengths, drive requests open-loop at increasing arrival rates so slow responses do not throttle the generator, and record TTFT and total latency percentiles plus error rate versus load to find the knee. Then run a soak test, and inject failures such as killing a replica. Fixed-length synthetic prompts and closed-loop generators flatter the system and produce capacity numbers that fail in production.
Follow-up: You have two zones and must survive a zone loss without violating the SLO. What does that cost?
A: Each zone must be able to carry the entire peak load alone, so total capacity is twice the requirement (100% over-provisioned); with three zones each carries half after losing one, so 1.5x. Alternatives are a degraded mode (shed low-priority traffic, route to a smaller model or another provider) or an explicitly weaker SLO for zone-loss events; the choice is a cost versus reliability decision that the product owner should sign off.
Follow-up: An incident produced 5% errors for 20 minutes. How much of a 99.9% monthly budget is that?
A: The burn rate is 5% / 0.1% = 50x. Twenty minutes is about 0.046% of the 43,200-minute window, and 0.046% x 5% = 0.0023% of all requests failed, against 0.1% allowed, so roughly 2.3% of the budget (assuming uniform traffic across the window). That is small in isolation, but at that burn rate the full budget would be gone in about 14 hours, which is why it should have paged.
Follow-up: Your load test says one GPU handles 2,000 tokens/s, yet production saturates at half that. Why might that be?
A: Common reasons: production prompts are longer (more prefill compute and more KV cache per sequence, so fewer concurrent sequences fit), output lengths are more variable, batching is less efficient because arrivals are sparser or more bursty, and sampling, tokenisation, networking and logging overhead were absent in the benchmark. The lesson is to benchmark with production-derived traffic and treat any vendor or published throughput figure as an upper bound for a specific setup.
In a design interview, capacity planning is the estimation step that justifies your architecture. For an LLM chat service, state the assumptions (say 10M requests per day, 4x peak, 500 output tokens, a specific model and an assumed per-GPU token rate you would benchmark), derive peak QPS, tokens per second, concurrency by Little's law, GPU count, and storage for logs and vector data, then connect each result to a decision: concurrency sets the batch and KV-cache budget and the connection pool sizes, tokens per second sets the GPU pool and its autoscaling signal, and a zone-loss requirement sets the replica floor (1.5x with three zones). Then tie it to the SLO: define TTFT and success-rate SLIs, an error budget, burn-rate alerts (see observability for AI systems), bounded queues with load shedding, and a degraded-mode plan (smaller model, cache, or provider fallback). Mention that GPU replicas take minutes to start, so autoscaling needs headroom or a warm pool, that batching raises throughput at the cost of per-request latency (batching and dynamic batching), and that costs are computed as GPUs x hours x rate with current pricing checked, then reduced by caching, quantisation and off-peak scale-down. Always present the numbers as estimates with assumptions and say how you would validate them with a load test.
Every reliability and latency target is bought with spare capacity: lower utilisation, more zones, more replicas, a warm pool. The decision is where on that curve the product belongs, and the cost is idle GPUs that produce nothing on a normal day. Tight SLOs are the wrong choice when users do not need them (internal tools, batch pipelines, best-effort features), because you pay for nines nobody notices and slow the team with an error budget it cannot realistically hold. Loose ones are the wrong choice for interactive, revenue-critical paths where a slow first token loses the user. Sizing from averages is the classic mistake (traffic is bursty and tails dominate), as is trusting a benchmark number from a different prompt mix, and as is treating capacity as a one-off: model upgrades, longer prompts and new features change per-request cost, so the plan needs a regular review, monitoring of budget burn and utilisation, and a rehearsed failure and cold-start path. When in doubt, choose a simple, explainable model, state its assumptions, and validate it with a measured test rather than a more elaborate but unverified one.