LLM inference is autoregressive text generation: the model reads a prompt, then emits one token at a time, each token conditioned on everything before it. That single fact explains almost every latency and cost curve you will be asked about: why output tokens are slower and pricier than input tokens, why GPU memory (not just GPU compute) caps how many users one machine can serve, and why "just use a bigger context window" is a capacity decision, not a free feature. Interviewers probe it because it separates people who call an LLM API from people who can size, price and debug the service behind it. For a Backend or AI engineer moving toward serving, gateways and RAG systems, this is the vocabulary (tokens, prefill, decode, KV cache, TTFT, ITL) that every later topic in this section builds on: batching, quantization, GPU utilization, streaming and cost optimization.
LLM inference is what happens after a language model has already been trained: you send it some text (a prompt) and it writes a reply back, one word-piece at a time, each new piece chosen based on everything written so far. Training builds the model once, offline, over weeks; inference is the moment-to-moment work of actually answering a real user's question, and it happens millions of times a day for a popular product. Every time you chat with an AI assistant, ask it to summarize a document, or get an auto-generated reply suggestion, you are triggering inference, and the speed, cost and reliability of that step is what backend and AI engineers are hired to manage.
Think of a translator working at a live event who has already spent years learning both languages (that's the training). During the event itself, the translator doesn't get the whole speech in advance — they listen to one phrase, think for a moment, and speak the translated version, then do it again for the next phrase, continuously, in real time. Inference is that live, phrase-by-phrase translation work: it depends on preparation done long ago, but the actual performance — with its own speed limits, effort, and cost per minute — happens fresh every single time someone needs it.
Models do not see characters or words; they see integer ids from a fixed vocabulary of subword pieces. Most production tokenizers are byte-pair-encoding (BPE) or a close relative (SentencePiece, byte-level BPE): start from single characters or bytes, repeatedly merge the most frequent adjacent pair in a training corpus, and record the merge order. Encoding replays those merges. Below is a toy trainer, run on a deliberately tiny corpus, so you can see why common words become one token and rare strings shatter.
corpus: "low low low low lower lower newest newest newest widest widest tokenization tokens tokenizer token"
word tokens (toy BPE) count
low ['low_'] 1 <- frequent word: one token
lowest ['low', 'est_'] 2 <- assembled from two learned pieces
newest ['newest_'] 1
tokenizing ['tokeniz','i','n','g','_'] 5 <- prefix learned, suffix never seen: falls back to characters
xyzzy ['x','y','z','z','y','_'] 6 <- unseen string: one token per character (real byte-level BPE falls back to bytes)The consequences are practical. Cost and context limits are counted in tokens, not characters, and the characters-per-token ratio depends on the tokenizer and the language: English prose is often quoted at roughly 3 to 4 characters per token, but code, JSON with long keys, non-Latin scripts, numbers and unusual identifiers can be much worse, so the same "1,000 words" may cost noticeably more in one language or format than another. Always count with the tokenizer of the exact model you call, because two providers' tokenizers give different counts for the same text. Tokenization also explains classic quirks: models are bad at counting letters in a word or reversing a string because they never see the letters, and leading spaces or casing change the token ids, so a prompt that differs by one space is a different input.
Where it breaks at scale: token counts drive both the bill and the latency, so anything that inflates them (verbose JSON schemas repeated in every request, whole documents pasted into every turn, multi-turn chat history re-sent each call) is a cost multiplier. The fix is engineering discipline (trim history, summarise, cache stable prefixes), not a better model.
ILLUSTRATIVE: 7B-parameter model, fp16 weights (14 GB), prompt = 2000 tokens
PREFILL (whole prompt, 1 pass) DECODE (1 token per pass)
FLOPs ~ 2 * P * n = 2*7e9*2000 = 2.8e13 ~ 2 * P = 1.4e10
bytes moved ~ 14e9 (weights read once) ~ 14e9 (weights read AGAIN) + KV cache reads
FLOP per byte ~ 2000 ~ 1 (batch of 1)
bound by compute (tensor cores busy) memory bandwidth (cores mostly idle)
Roofline test: ridge = peak FLOPS / peak bandwidth
hypothetical accelerator: 200 TFLOP/s and 2 TB/s -> ridge = 100 FLOP per byte
decode at batch 1 = ~1 FLOP/byte << 100 -> memory-bound
decode at batch B = ~B FLOP/byte (the same weight read is shared by B sequences)Generation has two phases. In prefill the model processes all prompt tokens in parallel in one forward pass; each weight matrix is loaded once and applied to thousands of token vectors, so the arithmetic per byte fetched is high and the GPU is compute-bound. It also fills the KV cache for the prompt and produces the first output token. In decode each step handles a single new token per sequence, but must still stream every weight (and the sequence's whole KV cache) from GPU memory to the compute units. The arithmetic per byte is now roughly one FLOP per byte, far below the accelerator's ridge point, so the step time is set by memory bandwidth, not by FLOPs. That is what "decode is memory-bound" means: a step cannot finish faster than bytes-moved divided by bandwidth (here 14 GB / 2 TB/s = 7 ms), no matter how many cores are idle.
Zooming into a single decode step shows where the bytes go, and why the KV cache term grows while the weight term does not:
one decode step, one sequence with n tokens of history (repeated for every layer)
new token x --> [ W_q, W_k, W_v projections ] reads layer weights (fixed size)
| |
| +--> append this token's K,V to the cache writes 2*kv_heads*head_dim*bytes
v
q . K_cache[0..n] -> softmax -> weights . V_cache[0..n] reads the WHOLE KV cache (grows with n)
|
v
[ W_o, MLP up/down ] reads layer weights (fixed size)
bytes per step ~ weight_bytes + batch * n * kv_bytes_per_token
^ constant ^ grows with context and with concurrencyThis one idea produces several levers. Batching many sequences shares one weight read across all of them, so throughput rises almost linearly until you hit compute or KV-memory limits. Quantizing weights (int8 or 4-bit) shrinks bytes per step and therefore step time. Speculative decoding spends the idle compute to verify several drafted tokens per weight read. Where it breaks: at long contexts the KV cache reads become comparable to the weight reads, and prefill of very long prompts is quadratic in attention cost, so neither phase is cheap forever.
Why a cache: attention for the newest token needs the Key and Value vectors of ALL earlier tokens.
Recomputing them each step costs O(n^2) total work over a generation; storing them costs O(n) memory.
bytes per token = 2 (K and V) * layers * kv_heads * head_dim * bytes_per_element
ASSUMPTIONS (illustrative shapes, check the real model card):
A) 7B-class, full multi-head attention: 32 layers, 32 KV heads, head_dim 128, fp16 (2 bytes)
B) 70B-class, grouped-query attention (GQA): 80 layers, 8 KV heads, head_dim 128, fp16
A) 2 * 32 * 32 * 128 * 2 = 524,288 B = 512 KiB per token
4,096-token sequence = 2.0 GiB 32,768-token sequence = 16.0 GiB
B) 2 * 80 * 8 * 128 * 2 = 327,680 B = 320 KiB per token (whole model, before sharding across GPUs)
4,096-token sequence = 1.25 GiB 32,768-token sequence = 10.0 GiB
int8 KV cache halves both (256 / 160 KiB per token).
How many concurrent 4k sequences fit? (80 GiB GPU, 14 GiB weights, 10% reserved for activations/fragmentation)
free = 80*0.9 - 14 = 58 GiB; per sequence = 2 GiB (config A) -> 29 sequencesThe KV cache grows linearly with sequence length and with the number of concurrent sequences, and it lives in scarce GPU memory next to the weights. So on a 7B-class model, weights are a fixed 14 GiB but each 4k-token user costs 2 GiB more: concurrency, not compute, is often what limits how many users a GPU can hold. That is why GQA and multi-query attention (fewer KV heads, here 4x smaller than full multi-head), KV-cache quantization, and paged allocation (vLLM's PagedAttention stores the cache in fixed-size blocks so memory is not wasted on over-reserved, fragmented contiguous buffers) are central serving techniques. Prefix caching goes one step further and reuses cache blocks for prompts that share a common start (for example the same system prompt).
Where it breaks: one 32k-token conversation in configuration A consumes 16 GiB, about as much as the model weights, so long-context requests displace many short ones; a service mixing both needs admission control, per-tenant limits, or separate pools. When memory runs out, engines either queue the request, preempt (swap out or recompute) another sequence, or fail with out-of-memory, and every one of those shows up as a latency spike.
request timeline
|-- queue --|---- prefill ----|-- decode step --|-- decode step --| ... |-- last token --|
submit first token appears done
|<---------- TTFT ----------->|<-- ITL ------->|<-- ITL ------->|
total latency = TTFT + (output_tokens - 1) * ITL
TTFT time to first token = queueing + prefill (grows with prompt length and load)
ITL inter-token latency = one decode step (per user; 1/ITL = the user's tokens/s)
Throughput = total tokens/s across ALL users on the machine (what your cost per token depends on)
Batch-size tension (ILLUSTRATIVE model in the demo below, ctx = 1000 tokens each):
batch ITL per user total tok/s batch ITL per user total tok/s
1 7.3 ms 138 32 15.4 ms 2079
8 9.1 ms 879 64 23.8 ms 2692These three numbers measure different things and optimizing one usually costs another. A chat product cares about TTFT (does it start answering quickly?) and ITL (does it stream faster than people read?). A nightly batch job cares only about throughput and will happily accept a high ITL. Raising the batch size increases total throughput (the weight read is amortised) but each step also reads more KV cache, so per-user ITL degrades: in the table, going from batch 1 to 64 gives roughly 20x the aggregate tokens per second while each user's stream slows by about a factor of three. Ideal batch size is therefore set by an SLO ("p95 ITL under 40 ms"), not by "as big as fits".
Batching policy matters as much as batch size, and a picture shows why. Each row is a slot; each character is one decode step; "." means the slot is idle.
static batching (batch of 4 runs until the LONGEST member finishes)
slot 0: AAAAAAAAAA............ A, B, C are short; D is long
slot 1: BBBB.................. B finishes at step 4 but its slot idles
slot 2: CCCCCC................ until D is done at step 24
slot 3: DDDDDDDDDDDDDDDDDDDDDDDD only then does the next batch start
continuous batching (slots refilled between decode steps)
slot 0: AAAAAAAAAAEEEEEEEEEEEEEE E, F, G were waiting; each joins as soon as a slot frees
slot 1: BBBBFFFFFFFFFFFFFFFFFFFF
slot 2: CCCCCCGGGGGGGGGGGGGGGGGG
slot 3: DDDDDDDDDDDDDDDDDDDDDDDDStatic batching holds a group until its slowest member finishes; short requests wait on long ones. Continuous (iteration-level) batching, used by vLLM, TGI and TensorRT-LLM among others, admits new sequences and retires finished ones between decode steps, keeping the hardware full. In the demo's 12-request simulation with 4 slots, continuous batching finishes everything at tick 520 versus 880, with mean completion 142 versus 587. Where it breaks: a long prompt's prefill can occupy a step and stall everyone else's decode (an ITL spike). Chunked prefill, or separating prefill and decode onto different workers ("disaggregated serving"), are the usual mitigations, at the price of more complexity and KV-cache transfer.
At each step the model outputs a score (logit) per vocabulary token; sampling parameters only transform those scores into the next token. They do not change the model, the KV cache or the compute, so they do not make generation faster or slower.
@@PY sampling.pyTemperature divides the logits before softmax: low values sharpen the distribution toward the top token (T=0.2 gives "Paris" 99.9%), high values flatten it (T=2 gives the nonsense token "zebra" 2.2% and "banana" 4.8%). Top-k keeps only the k most likely tokens; top-p (nucleus) keeps the smallest set whose probability mass reaches p, so it adapts to how peaked the distribution is. Temperature 0 is normally implemented as greedy argmax, and even then outputs are generally not guaranteed bit-identical across runs, because batching changes floating-point summation order (behaviour depends on the provider and framework). Do not build correctness on "temperature 0 is deterministic".
Context window = prompt + output must fit in the model's maximum length (check current docs; it varies by model).
cost of a long prompt:
compute : prefill work grows with prompt tokens (attention part grows ~ n^2) -> higher TTFT
memory : KV cache grows ~ n per sequence (2 GiB at 4k, 16 GiB at 32k in config A) -> lower concurrency
money : providers generally bill per input token and per output token, usually at different rates
ILLUSTRATIVE spend (PLACEHOLDER prices, not real): $1 per 1M input tokens, $4 per 1M output tokens
1M requests/day * (3,000 in, 500 out)
input : 3e9 tokens = 3,000 M * $1 = $3,000/day
output : 5e8 tokens = 500 M * $4 = $2,000/day -> 40% of spend from 14% of the tokensA larger window is a ceiling, not a target. Filling it costs money on every call, raises TTFT, shrinks how many concurrent users a GPU can hold, and generally does not improve answer quality linearly, since models can use information in the middle of a very long prompt less reliably than information near the start or end (model-dependent; evaluate on your task). Retrieval, summarisation and prompt caching exist precisely to send fewer, better tokens. See RAG and semantic caching.
Two stdlib-only programs. The first is a toy BPE tokenizer and a KV-cache calculator. The shapes are assumed, not measured, so swap in the real numbers from a model card.
@@PY tok_kv.pyThe second is a roofline-style latency model. The hardware (2 TB/s, 200 TFLOP/s) and the 7B model are hypothetical and the arithmetic ignores real-world overheads (kernel launch, attention's n^2 term, interconnect, imperfect bandwidth utilisation), so use it to reason about shape and ratios, never as a benchmark. It also simulates static versus continuous batching.
@@PY latency.pyThings to notice in the output. Batch-1 decode is "memory-bound" at 7.26 ms per token, and going to batch 64 multiplies total throughput roughly 19x while ITL per user only rises from 7 ms to 24 ms. With full multi-head attention the KV cache, not compute, stops the batch (118 sequences), and GQA lifts that to 475. Only with tiny contexts does the model become compute-bound. The same tokens in reverse (500 in, 2000 out) take about 3.7x longer than 2000 in, 500 out. Real serving code (vLLM, TensorRT-LLM, TGI) cannot be run here, so nothing below is measured output:
# vLLM-style server, illustrative flags only (verify against the docs for your version)
# --max-model-len 8192 cap context per request, bounds worst-case KV per sequence
# --gpu-memory-utilization 0.90 fraction of GPU memory the engine may use (weights + KV cache)
# --max-num-seqs 128 upper bound on concurrent sequences per step
# --enable-prefix-caching reuse KV blocks for shared prompt prefixes
# Then load-test with your real prompt/output length distribution and read TTFT, ITL and tokens/s.Q: Walk me through what happens from sending a prompt to receiving the first token.
A: The text is tokenized, queued until the scheduler has a slot and KV-cache space, then prefill runs all prompt tokens through the model in one parallel pass, stores their keys and values, and samples the first token. TTFT is queue time plus prefill time, so it grows with prompt length and with load. Everything after that is decode, one token per step.
Q: Why is decode memory-bound while prefill is compute-bound?
A: Prefill applies each loaded weight to thousands of prompt tokens, so it does thousands of FLOPs per byte fetched. Decode applies each weight to one token per sequence, roughly one FLOP per byte, far below a modern accelerator's ridge point (on the order of 100 FLOP/byte in my hypothetical example), so the cores wait on memory. A 14 GB model on 2 TB/s memory cannot decode faster than about 7 ms per token at batch 1, whatever the FLOPs.
Q: How do you compute KV cache size, and why does it matter?
A: 2 * layers * kv_heads * head_dim * bytes per element, per token, per sequence. For an assumed 7B-class model with 32 layers, 32 KV heads, head_dim 128 and fp16 that is 512 KiB per token, so a 4k-token sequence holds 2 GiB. On an 80 GiB GPU with 14 GiB of weights that is roughly 29 concurrent sequences, which is why KV memory, not FLOPs, often limits concurrency and why GQA (fewer KV heads) and cache quantization matter.
Q: Why do providers price output tokens above input tokens, and why does output length dominate latency?
A: Input tokens are processed in one parallel, compute-efficient prefill; each output token needs its own sequential, bandwidth-bound decode step that cannot be parallelised across the same sequence. In my model, 2000 in and 500 out takes about 3.9 s while 500 in and 2000 out takes about 14.3 s. Pricing ratios are provider decisions (check current docs), but the cost asymmetry follows from that difference.
Q: TTFT, inter-token latency, throughput: which do you optimise for a chatbot versus a nightly summarisation job?
A: A chatbot optimises TTFT and ITL, which means smaller batches, prefix caching and headroom to avoid queueing. A batch job optimises aggregate tokens per second and cost, so it runs the largest batch that fits and tolerates a large ITL. Mixing both on one pool means interactive traffic suffers when batch traffic fills the batch, so separate pools or priority classes are common.
Q: Does temperature 0 give deterministic output?
A: It selects the highest-probability token (greedy), so it removes sampling randomness, but the logits themselves can differ slightly between runs because floating-point summation order changes with batch composition and kernels, which can flip near-ties. So treat it as "mostly repeatable", cache by input if you need identical replies, and never depend on bit-exactness. Temperature and top-p change only the distribution, not speed.
Q: Static versus continuous batching: what is the difference and what does it buy?
A: Static batching waits for the slowest sequence, so short requests idle behind long ones. Continuous batching swaps sequences in and out at every decode step, keeping slots full. In my 12-request simulation it cut the finish time from tick 880 to 520 and mean completion from 587 to 142; the real gain depends on how variable output lengths are, and it needs a paged KV cache to allocate memory dynamically.
Q: A vendor advertises a huge context window. Why not always fill it?
A: Every call pays for the tokens in price and prefill time, the KV cache per sequence grows (16 GiB at 32k in my 7B-class example), which cuts concurrency, and answer quality over very long prompts is not uniform (model-dependent, so test it). Retrieving only the relevant passages, summarising history and caching stable prefixes usually give better cost and latency.
Follow-up: GQA reduced KV size 4x. Does it make decode compute-bound?
A: No. It reduces the KV bytes read and stored per token, which lifts concurrency (in my model from 118 to 475 sequences at 1000 tokens of context) and cuts per-step traffic, but the weight read still dominates at small batch and the arithmetic per byte is still low. The bound moves toward compute only at large batch sizes or short contexts, as the ctx=100 rows showed.
Follow-up: You int8-quantised the weights and decode sped up by less than 2x. Why?
A: Halving the weight bytes halves only that part of the memory traffic. KV cache reads (still fp16 unless quantised too), dequantisation overhead, attention kernels, sampling and launch overhead do not shrink, and the speedup shrinks as batch and context grow. Quantization also costs some accuracy, so validate on your evaluation set.
Follow-up: p99 inter-token latency spikes whenever a long-prompt request arrives. Diagnose and fix.
A: A long prefill occupies the GPU for many milliseconds and stalls the decode steps of every other running sequence. Options: chunked prefill (split the prompt so decode steps interleave), separate prefill and decode workers (disaggregation, which costs KV-cache transfer and complexity), a cap on prompt length or a separate pool for long-context traffic, and admission control so a burst of long prompts cannot fill the batch.
In an LLM API gateway or RAG chatbot design, inference fundamentals turn a vague "we need to serve N users" into numbers. A typical estimation, with stated assumptions: 1M requests per day is about 12 requests per second on average and perhaps 3x that at peak; if each request averages 3,000 input and 500 output tokens, that is 3.5B tokens per day, of which the 500M output tokens need sequential decode. If one GPU decodes an aggregate of a few thousand tokens per second at your target ITL (measure it, do not trust my illustrative figure), the output work alone is a handful of GPUs at peak, plus headroom for the KV cache, which is what actually limits the number of concurrent sequences per device. Latency budgets then split into queue plus prefill (TTFT), ITL times output length, and network/streaming overhead, which is why streaming is nearly mandatory for perceived latency. The same model drives autoscaling (scale on queue depth or KV-cache utilisation rather than CPU), SLOs (separate TTFT and ITL targets), and gateway policy such as per-tenant token budgets and max-token limits, because a request's true cost is input tokens plus output tokens times decode step time.
The core decision is how to spend a fixed GPU: larger batches and longer contexts raise throughput and lower cost per token, but every increment adds per-user latency, KV-cache pressure and tail risk. The knobs each have a price. Quantization saves memory and bandwidth at a possible quality cost; GQA-style architectures shrink the KV cache but are a model choice you inherit; speculative decoding trades spare compute and extra complexity for lower latency and helps least when the machine is already compute-saturated at high batch; prefix caching helps only when prompts share long, stable prefixes; disaggregated prefill/decode isolates the two workloads at the cost of moving KV cache between machines. Chasing tokens per second is the wrong goal for an interactive product, where p95 TTFT and ITL are the contract, and chasing latency is the wrong goal for offline work, where cost per token is. The mistake to avoid is treating "context window" and "max tokens" as free parameters: they are capacity and money, and they should be capped per tenant and per route.