Serving a model is a hardware-budgeting problem. A GPU is a very expensive machine with three separate limits: how many arithmetic operations it can do per second, how many bytes it can move per second from its main memory, and how many bytes that memory can hold at all. Almost every serving decision (batch size, quantization, tensor parallelism, which instance type to rent) is really a decision about which of those three limits you are hitting. Interviewers probe this because "the GPU is at 100% but throughput is bad" and "we need bigger GPUs" are the two most common wrong diagnoses in ML infrastructure, and because cost per token is the number that decides whether an AI product is viable. For a Backend or AI engineer moving toward inference work, you do not need CUDA expertise, but you must be able to do the arithmetic: does it fit, what is the bottleneck, what does a token cost.
Every hardware number on this page is ILLUSTRATIVE: a hypothetical accelerator with round figures chosen so the arithmetic is easy to follow, not a specification of any real product. Real peak FLOPs, bandwidth, and capacity differ by vendor, generation, and precision, and change quickly; check the vendor datasheet and measure on your own workload.
A GPU (graphics processing unit) is a chip built to do huge numbers of simple calculations at once, which happens to be exactly what running an AI model requires. "Utilization" is just a measure of how much of that chip's capacity is actually being put to work at a given moment, versus sitting idle waiting for data or for the next task. A GPU that is poorly utilized is like a factory with all its machines switched on but only half of them actually producing anything & the electricity bill is the same either way, so the goal is always to keep as much of the chip's capacity working on useful output as possible.
Think of a busy restaurant kitchen with several stoves. If a chef is only using one burner while the other five sit empty, the kitchen looks "open for business" but is nowhere near its real cooking capacity & orders back up even though nothing looks broken. Getting good GPU utilization is the same problem as a head chef's job: batching multiple orders so every burner is doing something useful at once, rather than lighting a stove for a single order and letting the rest go cold.
CPU: few, fast, flexible cores GPU: thousands of simple lanes, grouped
+------+ +------+ +------+ +------+ +-------------------------------+
| core | | core | | core | | core | | SM SM SM SM SM SM SM ...| SM = streaming multiprocessor
+------+ +------+ +------+ +------+ | (each: many lanes + tensor |
big caches, branch prediction, | cores + registers + shared |
out-of-order, ~10s of cores | memory/L1) |
DRAM ~ 100s of GB/s +---------------+---------------+
| L2 cache (tens of MB, illustrative)
Good at: branchy logic, tokenizing, |
parsing, orchestration, small work +---------------+---------------+
| HBM (high-bandwidth memory) |
| ~10s-100s of GB capacity |
| ~TB/s bandwidth (illustrative)|
+-------------------------------+
Approximate ORDER of magnitude, fastest/smallest to slowest/largest (illustrative ranks, not specs):
registers (per lane) > shared memory / L1 (per SM, KBs-MBs) > L2 (tens of MB) > HBM (tens of GB) > host DRAM over PCIe > SSD/networkNeural network inference is dominated by dense matrix multiplication, which is enormous amounts of identical, independent multiply-adds. GPUs win because they spend their transistors on arithmetic lanes (and dedicated matrix units, often called tensor cores) rather than on the branch prediction and large per-core caches a CPU needs for arbitrary code. CPUs still matter in a serving stack: tokenization, request parsing, sampling logic in some stacks, and scheduling all run on the host, and a slow host thread can starve the GPU.
The memory hierarchy is the part interviewers care about. The weights of a large model live in HBM (high-bandwidth memory), a large off-chip pool on the GPU package. The actual arithmetic happens on-chip, in units fed from small, very fast on-chip memories (registers, shared memory, L2). Every byte of weight used in a matmul must travel HBM to chip, and HBM bandwidth, while high, is finite. The design of fast kernels (FlashAttention is the well-known example) is mostly about reusing data while it is in fast on-chip memory so it is not re-read from HBM. Moving data off the GPU (host DRAM over PCIe, or across the network) is far slower again, which is why models are loaded once and kept resident rather than paged per request.
Arithmetic intensity (AI) = FLOPs performed / bytes moved between HBM and chip
attainable FLOP/s = min( peak_FLOPs , AI x memory_bandwidth )
FLOP/s
^ peak (flat "roof")
| ______________________
| /
| / <- slope = memory bandwidth
| / (memory-bound: more FLOPs per byte helps)
| /
+-------/----------------------------> arithmetic intensity (FLOP/byte)
ridge point = peak_FLOPs / bandwidth
left of ridge: MEMORY-BOUND right of ridge: COMPUTE-BOUND
Matrix multiply C[M,N] = A[M,K] x B[K,N] in 16-bit:
FLOPs = 2*M*K*N bytes ~ 2*(M*K + K*N + M*N)
M = 1 (one token, decode) -> AI ~ 1 FLOP/byte (the weight matrix is read once per single token)
M = 512 (prompt, prefill) -> AI ~ hundreds (the same weights are reused across many tokens)The roofline model says a kernel is limited by whichever of the two ceilings it hits first. Below the ridge point (peak compute divided by memory bandwidth), the chip is waiting for bytes and the arithmetic units are mostly idle; above it, the arithmetic units are the limit. LLM decode is the textbook memory-bound case: to generate one token you must read essentially every weight (and the KV cache) from HBM to do a tiny amount of arithmetic per byte. Prefill processes many prompt tokens against the same weights at once, so the intensity is high and it tends to be compute-bound (see LLM Inference Fundamentals for the two phases). The bottleneck is a property of the workload shape, not of the model alone.
Batching is the lever that moves a workload to the right on this plot: running 64 sequences through the same weights reads the weights once but does 64 tokens of arithmetic, multiplying intensity by about 64 (see Batching & Dynamic Batching for why a batch is nearly free until the ridge point). Quantization moves it too, by shrinking the bytes (see Model Quantization & Compression). The limits: KV cache bytes grow with batch size and context, so a bigger batch also adds memory traffic and eats VRAM, and real kernels rarely reach either roof. The calculator in "Try It Yourself" puts numbers on all of this.
nvidia-smi "GPU-Util" (as documented): percent of the sample period during which
at least one kernel was executing on the GPU. It is a TIME metric, not an EFFICIENCY metric.
Scenario A: 1 tiny kernel running on 2 of 100+ SMs, 100% of the time -> GPU-Util = 100%
Scenario B: a big matmul on all SMs at high tensor-core throughput -> GPU-Util = 100%
(same number, 50x different work)
Metrics that say more (available via DCGM / profilers; names vary by tool and generation):
SM activity / occupancy how many SMs actually have work resident
tensor pipe activity are the matrix units busy?
DRAM (HBM) bandwidth util close to the roof? then memory-bound, batching/quantization helps
MFU (model FLOPs utilisation) achieved model FLOP/s divided by peak (a training/inference efficiency KPI)
power draw a crude but useful proxy: a truly busy GPU draws near its power limitThe number a dashboard shows as "utilisation" usually only says the GPU was not idle, and the naive reading, "100% means we are out of capacity, add GPUs", is frequently wrong. A memory-bound decode workload at batch size 1 can show a very high GPU-Util while using a few percent of peak FLOPs, and the correct fix is bigger batches, not more GPUs. The opposite trap also exists: a service can show 30% util because requests arrive in bursts with idle gaps, yet be perfectly healthy. Look at bandwidth utilisation, SM/tensor activity, queue depth, and end-to-end tokens per second per GPU, then compare against the roofline expectation. The batching page has a longer treatment of this trap under the latency-versus-throughput trade.
Where this breaks at scale: teams autoscale on GPU-Util and either over-provision (util looks high because kernels are inefficient) or under-provision (util looks low while the queue is growing). Autoscaling on queue depth, time-in-queue, or tokens per second is generally more reliable; see Autoscaling for Inference Workloads.
VRAM (usable) = weights + KV cache + activations/workspace + runtime overhead
weights = parameters x bytes_per_param 7B params x 2 bytes (FP16/BF16) = 14 GB (INT8: ~7 GB, INT4: ~3.5 GB)
KV cache / token = 2 (K and V) x layers x kv_heads x head_dim x bytes
KV cache total = KV/token x tokens_per_sequence x concurrent_sequences
activations = temporary tensors for the current batch (grows with tokens in flight; framework-dependent)
overhead = CUDA context, allocator fragmentation, graph/kernel workspaces (leave a margin, often ~5-15%)
Hypothetical 7B model, 32 layers, 8 KV heads (grouped-query attention), head_dim 128, FP16 KV:
KV/token = 2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes = 128 KiB
one 4,096-token sequence = 512 MiB of KV cache
with 32 KV heads (plain multi-head attention) the same model would need 4x: 2 GiB per 4k sequenceWeights are a fixed cost paid once per GPU (or per shard); the KV cache is the variable cost that scales with how many sequences are in flight and how long they are, which is what really limits concurrency. That is why a model that "fits" (weights below VRAM) can still serve very few users: the free memory after weights divided by the per-sequence KV size is your maximum concurrent sequences, and that number often decides the batch size, not compute. It also explains two design choices you will hear about: grouped-query attention (fewer KV heads means a smaller cache) and KV cache quantization or paging (vLLM's PagedAttention manages KV in blocks to avoid fragmentation waste). The full KV formula is derived in LLM Inference Fundamentals.
At scale the budget fails in ways that look random: an out-of-memory error appears only when several long prompts arrive together, fragmentation keeps a "free" gigabyte unusable, or a larger max context in config silently cuts concurrency. The safe habit is to budget for the worst-case configured context times the expected concurrency, leave a margin, and load-test with realistic length distributions rather than the average.
Tensor parallel (TP): split EACH layer's weight matrices across GPUs; partial results combined by all-reduce
GPU0 [W part 0] --\
GPU1 [W part 1] ---> all-reduce (~2 per layer) --> next layer needs FAST links (NVLink-class), latency-sensitive
Pipeline parallel (PP): split LAYERS into stages; activations flow stage to stage
GPU0 [layers 0-15] --> GPU1 [layers 16-31] lighter traffic, but pipeline bubbles and added latency
Data parallel (DP): full copy of the model per GPU; requests are spread across copies
GPU0 [full model] GPU1 [full model] no per-token communication; each copy must fit alone
(in TRAINING, DP also all-reduces gradients; for serving it is just "replicas behind a load balancer")Use the simplest option that works. If the model plus a useful KV cache fits on one GPU, run replicas (data parallel): it scales almost linearly, fails independently, and needs no fast interconnect. If it does not fit, you shard: tensor parallelism cuts per-GPU weight and KV memory and can also cut per-token latency, but every layer now needs collective communication, so it is generally kept inside one node connected by a high-bandwidth GPU-to-GPU link. Pipeline parallelism suits crossing nodes because it sends activations, not partial sums, but it adds per-request latency and needs enough concurrent work to keep all stages busy. Real deployments combine them (for example TP inside a node, PP or DP across nodes), and mixture-of-experts models add expert parallelism; the exact split depends on the framework (vLLM, TensorRT-LLM and others support different combinations, check current docs).
# Tensor-parallel communication per decode step (TP=4, 32 layers, d_model 4096, FP16, ~2 all-reduces/layer)
# Link numbers ASSUMED: NVLink-class 450 GB/s + 10 us latency; PCIe-class 25 GB/s + 20 us latency
# Output (real):
# batch= 1: payload 8 KiB x 64 all-reduces
# NVLink-class (450 GB/s, 10 us lat): 0.64 ms comm per decode step
# PCIe-class ( 25 GB/s, 20 us lat): 1.31 ms comm per decode step
# batch= 64: payload 512 KiB x 64 all-reduces
# NVLink-class (450 GB/s, 10 us lat): 0.75 ms comm per decode step
# PCIe-class ( 25 GB/s, 20 us lat): 3.29 ms comm per decode step
# batch=512: payload 4096 KiB x 64 all-reduces
# NVLink-class (450 GB/s, 10 us lat): 1.53 ms comm per decode step
# PCIe-class ( 25 GB/s, 20 us lat): 17.39 ms comm per decode stepRead the numbers as a shape, not a benchmark. At small batch the communication cost is dominated by fixed latency times the number of collectives (64 per step here), which is why TP is latency-sensitive even with fast links. At larger batch the payload grows and a slow link (PCIe-class here) makes communication a large share of the step time (17 ms per step in the last row, compared with the roughly 67 microseconds per large matmul in the roofline output above), which is why TP across a slow interconnect is generally a poor choice. The model is deliberately simple (ring all-reduce volume, no overlap of communication with compute, no contention), so treat it only as a reason to prefer fast links inside a TP group.
cost per 1M generated tokens = ($ per GPU-hour / 3600) / (tokens per second per GPU) x 1,000,000
ILLUSTRATIVE only: assume $2.50 per GPU-hour (a placeholder, not a quote; prices vary widely by provider,
region, commitment and GPU generation)
throughput 750 tok/s -> 2.50/3600/750 x 1e6 = $0.93 per 1M tokens
throughput 3,000 tok/s -> 2.50/3600/3000 x 1e6 = $0.23 per 1M tokens (4x throughput = 4x cheaper per token)
The GPU costs the same whether it is idle or saturated. Cost per token is therefore mostly a UTILISATION story.An inference GPU is rented by the hour (or owned and depreciated), so its cost is roughly fixed; what you control is how many useful tokens it produces per second. That is why batching, quantization, KV-cache efficiency and fewer idle GPUs are cost levers, and why a raw "dollars per GPU-hour" comparison between instance types is misleading without tokens per second per dollar measured on your own model and traffic. It also means reserved or committed capacity beats on-demand only if you keep it busy; bursty workloads may prefer scaling to lower counts or hosted per-token APIs. The wider picture is in Cost Optimization for AI Systems and Multi-Model & Multi-Tenant Serving, where GPU sharing is used to raise utilisation.
Two stdlib-only calculators for a hypothetical accelerator (300 TFLOP/s dense FP16, 2.0 TB/s HBM, 80 GB) and a hypothetical 7B model (32 layers, 8 KV heads, head_dim 128, 4,096-token sequences, $2.50 per GPU-hour). All inputs are assumptions; the outputs are real results of running this code, but they describe the assumed hardware, not any product. First, a roofline for one 8192x8192 matmul:
# roofline.py
PEAK_FLOPS = 300e12 # 300 TFLOP/s dense FP16/BF16 (assumed)
HBM_BW = 2.0e12 # 2.0 TB/s HBM bandwidth (assumed)
RIDGE = PEAK_FLOPS / HBM_BW # FLOP per byte where the two limits meet
def gemm(M, K, N, bytes_per=2):
"""C[M,N] = A[M,K] @ B[K,N]; count each matrix moved once between HBM and chip."""
flops = 2 * M * K * N
moved = bytes_per * (M * K + K * N + M * N)
ai = flops / moved
attainable = min(PEAK_FLOPS, ai * HBM_BW)
t = max(flops / PEAK_FLOPS, moved / HBM_BW)
bound = "compute" if ai >= RIDGE else "memory"
return ai, attainable, t, bound
print(f"ridge point = {RIDGE:.0f} FLOP/byte (peak {PEAK_FLOPS/1e12:.0f} TFLOP/s, HBM {HBM_BW/1e12:.1f} TB/s)")
print(f"{'workload':34} {'AI':>7} {'TFLOP/s':>8} {'%peak':>6} {'time us':>9} bound")
K = N = 8192
cases = [("decode, batch=1", 1), ("decode, batch=8", 8), ("decode, batch=64", 64),
("decode, batch=256", 256), ("prefill, 512 tokens", 512), ("prefill, 4096 tokens", 4096)]
for name, M in cases:
ai, att, t, b = gemm(M, K, N)
print(f"{name:34} {ai:7.1f} {att/1e12:8.1f} {100*att/PEAK_FLOPS:5.1f}% {t*1e6:9.1f} {b}")
# Same weights, fewer bytes: INT8 weights (activations still FP16) at batch=1
flops = 2 * 1 * K * N
moved_fp16 = 2 * (K + K * N + N)
moved_int8 = 2 * K + 1 * K * N + 2 * N
print(f"\nbatch=1 weight bytes: FP16 {moved_fp16/1e6:.0f} MB -> INT8 {moved_int8/1e6:.0f} MB; "
f"time {moved_fp16/HBM_BW*1e6:.1f} us -> {moved_int8/HBM_BW*1e6:.1f} us (memory-bound: time follows bytes)")
# Output:
# ridge point = 150 FLOP/byte (peak 300 TFLOP/s, HBM 2.0 TB/s)
# workload AI TFLOP/s %peak time us bound
# decode, batch=1 1.0 2.0 0.7% 67.1 memory
# decode, batch=8 8.0 16.0 5.3% 67.2 memory
# decode, batch=64 63.0 126.0 42.0% 68.2 memory
# decode, batch=256 240.9 300.0 100.0% 114.5 compute
# prefill, 512 tokens 455.1 300.0 100.0% 229.1 compute
# prefill, 4096 tokens 2048.0 300.0 100.0% 1832.5 compute
#
# batch=1 weight bytes: FP16 134 MB -> INT8 67 MB; time 67.1 us -> 33.6 us (memory-bound: time follows bytes)Note batch 1 to batch 64: the matmul takes essentially the same 67-68 microseconds, yet does 64 times the useful work. That is the "batch is nearly free" effect, and it ends near the ridge point. Now the VRAM budget and a crude cost estimate. The throughput line assumes every decode step reads all weights plus the full-context KV cache of every running sequence and ignores compute, attention overheads, scheduling and latency limits, so it is an upper-bound-style illustration, not a prediction:
# vram.py
import math
GIB = 1024**3
GPU_GB = 80 # capacity (treated as 80 GiB for simplicity)
HBM_BW = 2.0e12 # bytes/s, assumed
PRICE_HR = 2.50 # assumed $/GPU-hour, NOT a quote
PARAMS = 7.0e9
LAYERS, KV_HEADS, HEAD_DIM = 32, 8, 128 # GQA: 8 KV heads
CTX = 4096 # tokens per sequence (prompt + output), assumed
RESERVE = 0.10 # kept for allocator fragmentation / CUDA context
ACT_GIB = 2.0 # activation + workspace, assumed
def kv_per_token(kv_heads, bytes_per=2):
return 2 * LAYERS * kv_heads * HEAD_DIM * bytes_per # 2 = K and V
def budget(label, w_bytes, kv_heads, kv_bytes):
usable = GPU_GB * GIB * (1 - RESERVE)
weights = PARAMS * w_bytes
free = usable - weights - ACT_GIB * GIB
per_seq = kv_per_token(kv_heads, kv_bytes) * CTX
seqs = int(free // per_seq) if free > 0 else 0
step = (weights + seqs * per_seq) / HBM_BW
tok_s = seqs / step if seqs else 0
cost = PRICE_HR / 3600 / tok_s * 1e6 if tok_s else float("inf")
print(f"{label:30} weights {weights/GIB:5.1f} GiB | KV/token {kv_per_token(kv_heads, kv_bytes)/1024:5.0f} KiB"
f" | KV/seq {per_seq/GIB:4.2f} GiB | free {free/GIB:5.1f} GiB | max seqs {seqs:4d}"
f" | ~{tok_s:6.0f} tok/s | ~${cost:5.2f}/1M tok")
budget("FP16 weights, GQA(8), FP16 KV", 2, 8, 2)
budget("FP16 weights, MHA(32), FP16 KV", 2, 32, 2)
budget("INT8 weights, GQA(8), FP16 KV", 1, 8, 2)
budget("INT4 weights, GQA(8), FP16 KV", 0.5, 8, 2)
budget("INT4 weights, GQA(8), INT8 KV", 0.5, 8, 1)
w70 = 70e9 * 2
print(f"\n70B FP16 weights = {w70/GIB:.0f} GiB > one {GPU_GB} GiB card -> need >= {math.ceil(w70/(GPU_GB*GIB*(1-RESERVE)))} GPUs before any KV cache")
# Output:
# FP16 weights, GQA(8), FP16 KV weights 13.0 GiB | KV/token 128 KiB | KV/seq 0.50 GiB | free 57.0 GiB | max seqs 113 | ~ 3027 tok/s | ~$ 0.23/1M tok
# FP16 weights, MHA(32), FP16 KV weights 13.0 GiB | KV/token 512 KiB | KV/seq 2.00 GiB | free 57.0 GiB | max seqs 28 | ~ 755 tok/s | ~$ 0.92/1M tok
# INT8 weights, GQA(8), FP16 KV weights 6.5 GiB | KV/token 128 KiB | KV/seq 0.50 GiB | free 63.5 GiB | max seqs 126 | ~ 3376 tok/s | ~$ 0.21/1M tok
# INT4 weights, GQA(8), FP16 KV weights 3.3 GiB | KV/token 128 KiB | KV/seq 0.50 GiB | free 66.7 GiB | max seqs 133 | ~ 3551 tok/s | ~$ 0.20/1M tok
# INT4 weights, GQA(8), INT8 KV weights 3.3 GiB | KV/token 64 KiB | KV/seq 0.25 GiB | free 66.7 GiB | max seqs 266 | ~ 7102 tok/s | ~$ 0.10/1M tok
#
# 70B FP16 weights = 130 GiB > one 80 GiB card -> need >= 2 GPUs before any KV cacheTwo lessons in that output. Quantizing weights alone gives only a modest gain here (about 3,027 to 3,551 tok/s) because with 100+ concurrent sequences the KV cache traffic is larger than the weight traffic; shrinking the KV cache (fewer KV heads, or 8-bit KV) is what doubles or quadruples concurrency. And the "2 GPUs" for 70B is a lower bound from weights alone, so a real deployment usually needs more for KV cache and headroom.
Q: Why are GPUs better than CPUs for neural network inference?
A: Inference is dominated by dense matrix multiplication, which is many independent multiply-adds. GPUs devote their silicon to thousands of arithmetic lanes and matrix units and pair them with very high-bandwidth memory, so they deliver far more FLOP/s and bytes/s than a CPU for that workload. CPUs remain better for branchy, low-parallelism work and for very small models or low traffic, where a GPU's cost and transfer overhead are not justified.
Q: What does "memory-bound" mean, and why is LLM decode memory-bound?
A: A kernel is memory-bound when its arithmetic intensity (FLOPs per byte moved from HBM) is below the hardware ridge point, so the chip waits for data rather than computing. In decode, each new token needs all the weights and the KV cache read from HBM but does only about one multiply-add per weight per sequence. With the assumed hardware above (ridge 150 FLOP/byte) a batch-1 matmul runs at about 1 FLOP/byte, under 1% of peak compute. Larger batches reuse each weight read across more sequences and raise intensity.
Q: A GPU shows 100% utilisation in nvidia-smi but throughput is low. What do you check?
A: GPU-Util only reports the fraction of time at least one kernel was running, so it can be 100% for tiny inefficient kernels. I would look at SM and tensor-core activity, HBM bandwidth utilisation, batch size in flight, host-side stalls (tokenization, Python overhead, data copies), and tokens per second per GPU against a roofline expectation. Often the answer is a batch size far below the ridge point, so batching improves throughput without adding GPUs.
Q: How do you decide whether a 7B model fits on one 80 GB GPU and how many users it can serve?
A: Weights are about 14 GB in FP16 (parameters times 2 bytes). From the remaining VRAM, subtract a reserve (say 10%) and activation workspace, then divide by KV cache per sequence, which is 2 times layers times KV heads times head_dim times bytes times tokens. For a 32-layer, 8-KV-head, 128 head_dim model at 4k context that is about 0.5 GiB per sequence, giving on the order of 100 concurrent sequences; with 32 KV heads it drops to about 28. The assumptions (context length, reserve, KV precision) should be stated aloud.
Q: When do you use tensor parallelism versus pipeline parallelism versus just replicating the model?
A: Replicate (data parallel) when the model and a useful KV cache fit on one GPU, because it needs no fast interconnect and scales linearly. Use tensor parallelism when the model does not fit or single-request latency must drop; it needs collectives every layer, so it belongs inside a node with a fast GPU-to-GPU link. Use pipeline parallelism to span nodes or fit very large models with lighter traffic, accepting bubbles and extra latency.
Q: How would you estimate the cost per million tokens for a self-hosted model?
A: Take the GPU-hour price, divide by 3,600 for a per-second cost, and divide by measured tokens per second per GPU at your target latency; multiply by a million. With an assumed $2.50 per hour and 3,000 tok/s that is roughly $0.23 per million generated tokens. The catch is that the throughput must be measured under realistic traffic, and idle time (nights, bursts) raises the effective cost, so utilisation is the real variable.
Q: Why can adding more GPUs make a serving system slower or no better?
A: If the bottleneck is elsewhere (host CPU, network, a small batch size, a queueing policy), extra GPUs just idle. If you add them as a tensor-parallel group over a slow interconnect, communication can exceed the compute saved, as the PCIe-class rows in the communication table show. Adding replicas helps only when the current replicas are saturated at a good batch size and the load balancer can spread work.
Follow-up: Quantizing weights to INT4 didn't increase our throughput much. Why?
A: Quantization helps most when weight reads dominate memory traffic, i.e. small batches. At high concurrency and long context the KV cache reads and capacity dominate, and weight-only quantization does not shrink them, as the calculator's 3,027 to 3,551 tok/s shows. Also, dequantization overhead and kernel maturity can eat some of the gain. The levers then are KV cache size (GQA, KV quantization, shorter contexts) and scheduling.
Follow-up: Your model needs 130 GB of weights in FP16. What are your options?
A: Shard across at least two 80 GB GPUs with tensor parallelism (within a node with fast links) and budget extra for KV cache, or quantize to INT8 (about 65 GB) or INT4 (about 33 GB) so it fits on one card, accepting some accuracy risk that must be evaluated. Which is right depends on latency targets, cost per token, and how much quality loss the task tolerates; measure both.
Follow-up: What should an autoscaler for GPU inference scale on, if not GPU-Util?
A: Signals closer to user impact and real saturation: queue depth or time-in-queue, in-flight requests versus KV-cache capacity, and tokens per second per replica against a measured knee. GPU-Util is coarse and lagging, and GPU startup plus model load times (often minutes) mean you also need headroom or warm pools, so the scaling policy must anticipate load rather than react.
In an LLM-serving design, hardware arithmetic is how you turn a traffic estimate into a fleet size. As an ILLUSTRATIVE example, assume 1M requests per day averaging 500 output tokens, so 500M generated tokens per day, or roughly 5,800 tokens per second on average and perhaps 3 times that at peak. If one GPU sustains about 3,000 tokens per second at acceptable latency (the value from the calculator on assumed hardware, which you must replace with a load-test number for your model), average load needs about 2 GPUs and peak about 6, before adding redundancy across zones (typically N+1 or more) and headroom for slow scale-up. Then check the VRAM side: at 100 concurrent 4k-token sequences the KV cache alone is around 50 GiB on the assumed model, which is why concurrency, not compute, often sets the replica count. Multi-GPU sharding appears when the model exceeds one card, at which point node topology (NVLink-class within a node, network between nodes) becomes a scheduling constraint in Kubernetes. Also mention cost: at an assumed $2.50 per GPU-hour, a 6-GPU fleet running around the clock costs on the order of $10,800 per month, so 30% higher utilisation from batching or quantization is real money. Related designs: Model Serving Frameworks, Autoscaling for Inference Workloads, and Cost Optimization for AI Systems.
The core decision is how to spend a fixed, expensive resource across latency, throughput and cost. Larger batches and denser packing raise GPU efficiency and lower cost per token, but lengthen queueing, raise per-request latency, and consume VRAM through KV cache, so the right operating point is the largest batch that still meets the latency SLO, found by measurement. Bigger or more GPUs buy capacity but add cost, interconnect dependence and failure coupling (a tensor-parallel group fails as a unit); smaller quantized models on fewer GPUs are cheaper but risk quality loss. Renting on-demand keeps flexibility but costs more per hour than committed capacity, which is only cheaper if it stays busy. This whole approach is the wrong choice when traffic is tiny or highly bursty (a hosted per-token API avoids paying for idle GPUs), when a small model on CPU meets the latency need, or when you cannot measure: teams that size from datasheet peaks instead of load tests routinely mis-provision by large factors.