Batching & Dynamic Batching

Batching means running several requests through the model in one forward pass instead of one at a time. It exists because a GPU is badly underused by a single request: most of the time for a small batch goes into moving weights from memory and launching kernels, not into arithmetic, so extra rows in the batch are close to free until the hardware saturates. Interviewers probe it because it is the single biggest lever on cost per request in inference, and it is a pure trade-off question: every millisecond you wait to fill a batch is a millisecond added to somebody's latency. For a Backend or AI engineer the useful skill is explaining why throughput rises with batch size, what dynamic and continuous batching change, and where it breaks (padding waste, head-of-line blocking, memory limits, tail latency). It builds on LLM Inference Fundamentals and GPU Hardware & Utilization Basics.

What Is Batching?

Batching is grouping several requests together and processing them in one go, instead of handling each one separately from start to finish. It's a way of getting more work done with the same amount of effort, because a lot of the "setup" cost of doing the work is the same whether you're doing it for one item or several.

What Is Dynamic Batching?

Dynamic batching is deciding how to group requests on the fly, based on what's actually arriving right now, instead of using a fixed group size decided in advance. If requests are coming in quickly, it forms bigger groups; if they're trickling in slowly, it doesn't make everyone wait too long just to fill an arbitrary quota.

Where This Is Used

• AI services like ChatGPT or an image-generation API group many users' requests together on the same GPU to serve everyone more cheaply and quickly.
• Ride-share apps batch nearby pickup requests when matching drivers during high-demand periods.
• Payment processors batch small transactions together for settlement instead of processing each one as an isolated event.
• Print shops and warehouses batch similar print jobs or shipping orders together to run the machinery once instead of many times.

A Day-to-Day Example

Think of a bakery's oven. Baking one cookie at a time would waste almost all the oven's heat and time. Instead, the baker waits for a tray to fill up with cookies before putting it in — but not so long that the first cookies placed on the tray go stale waiting. Dynamic batching is the baker adjusting how long to wait based on how fast orders are coming in, rather than always waiting for a completely full tray.

Deep Dive: Why a Batch Is Nearly Free

One decode step of an LLM (or one forward pass of a small model), simplified:

  time ~= (bytes of weights read from GPU memory) / (memory bandwidth)     <- paid ONCE per pass
        + (FLOPs for the rows in the batch) / (compute throughput)          <- grows with batch size

ILLUSTRATIVE arithmetic (assumptions, not a measurement): a 7B-parameter model in fp16 is ~14 GB of weights.
On a GPU with ~1 TB/s of memory bandwidth, reading the weights once takes ~14 ms.

  batch = 1 :  14 ms per step ->  1 token per 14 ms
  batch = 8 :  ~14 ms per step ->  8 tokens per 14 ms   (compute is still far below the GPU's limit)
  batch = 64:  step time starts to grow as compute (and the KV cache) becomes the bottleneck

  cost per request  =  step time / batch size   -> falls sharply at first, then flattens

This is the roofline idea: at small batch sizes the workload is memory-bandwidth bound, so adding rows raises arithmetic intensity (useful FLOPs per byte read) at almost no extra time. That is why decode, which reads all the weights to produce one token per sequence, benefits so much, while prefill (processing the whole prompt at once) is already compute-heavy and gains less. The curve does not stay free forever: once compute or memory capacity becomes the limit, larger batches just make each step slower, and every request in the batch pays for it.

Where it breaks: memory. Each running sequence holds a KV cache of roughly 2 x layers x kv_heads x head_dim x bytes_per_value per token. For an illustrative 32-layer model with 32 KV heads, head_dim 128 and fp16, that is 2 x 32 x 32 x 128 x 2 = 524,288 bytes, i.e. 0.5 MiB per token, so a 2,000-token sequence holds about 1 GiB. The batch you can actually run is therefore bounded by GPU memory left after the weights, not just by compute. Models using grouped-query attention have far fewer KV heads and a proportionally smaller cache.

Deep Dive: Static vs Dynamic Batching, Max Batch Size and Max Wait

STATIC (fixed size, no timeout)             DYNAMIC (max_batch=8, max_wait=10 ms)

req1 arrives ----+                          req1 arrives -- start timer (10 ms)
req2 arrives ----+ waiting...               req2 arrives    (queued)
 ...             |                          timer fires or 8th request arrives
req8 arrives ----+ -> run batch of 8         -> run whatever is queued (1..8 requests)
Low traffic: req1 may wait seconds          Worst-case added wait is bounded by max_wait

Two knobs of a dynamic batcher:
  max_batch_size : upper bound; limited by GPU memory and by how slow a full batch is
  max_wait       : how long the OLDEST queued request may be held hoping for company
  Launch rule    : launch when (queue >= max_batch) OR (oldest waited >= max_wait) AND the GPU is free

Static batching (in the sense used for online serving) is simple and gives the best packing, but under variable traffic its latency is unbounded: at 20 requests/s a batch of 8 takes ~350 ms to fill on average, and if traffic stops the last few requests may never run without a flush timer. Dynamic batching adds the timer, so the queueing delay is capped at max_wait (plus time spent waiting for a busy GPU) while batches still grow when load is high, because requests pile up while the previous batch is running. That "batching for free under load" property is the key insight: you do not need to wait at all when the GPU is busy anyway.

Triton Inference Server exposes this as a per-model setting; the sketch below is config, not something run here, and field names should be checked against the current Triton docs:

# Triton model config.pbtxt (sketch, not executed)
max_batch_size: 16
dynamic_batching {
  preferred_batch_size: [ 4, 8 ]
  max_queue_delay_microseconds: 5000     # max_wait = 5 ms
}

Deep Dive: Padding Waste and Length Bucketing

Batch of 4 prompts padded to the longest (tokens):

  req A  |##########..............|   10 real, 14 wasted     Tensor is [4 x 24].
  req B  |##########################|  24 real                Real tokens = 10+24+8+12 = 54
  req C  |########................|   8 real, 16 wasted       Padded slots = 4 x 24  = 96
  req D  |############............|  12 real, 12 wasted       Waste = 42 / 96 = 43.75%

Fixes: sort/bucket requests by length before batching, cap max length, use ragged/packed
batches or attention kernels that skip padding, or (for LLMs) continuous batching with paged KV cache.

Batching only works if rows can share one tensor. Variable-length text is padded to the longest row, and the GPU spends compute on padding that is thrown away. The waste grows with batch size and with length variance, so bigger batches partly cancel their own benefit. Bucketing requests into length classes (for example 64/128/256 tokens) reduces waste but splits the queue, so each bucket fills more slowly and needs its own wait timer. Many modern serving stacks avoid the problem with packed or ragged representations; check what your framework does before assuming padding cost exists.

The simulation in "Try It Yourself" makes this visible: with static batches of 8 and lognormal-distributed input lengths, about half of all token slots were padding, while dynamic batching at moderate load formed smaller batches and padded less.

Deep Dive: Continuous (Iteration-Level) Batching and Head-of-Line Blocking

Generation is a loop: 1 prefill pass, then one decode pass per output token. Sequences finish at different times.

STATIC batch of 4 (whole batch runs until the longest sequence ends)
 slot 1  |A A A A A A A A A A A A|   long sequence
 slot 2  |B B B . . . . . . . . .|   finished at step 3, slot idle for 9 steps
 slot 3  |C C C C C . . . . . . .|
 slot 4  |D D . . . . . . . . . .|   result held back until step 12
          new requests must wait until step 12 to start   <- head-of-line blocking

CONTINUOUS batching (scheduler runs after EVERY decode step)
 slot 1  |A A A A A A A A A A A A|
 slot 2  |B B B E E E E E . . . .|   E joins the moment B finishes
 slot 3  |C C C C C F F F F F F F|
 slot 4  |D D G G G G G H H H H H|
          finished sequences leave immediately; waiting ones take the freed slot

Static batching is a poor fit for autoregressive generation because output lengths differ wildly: short requests are held hostage by the longest one in their batch (head-of-line blocking) and their slots sit idle. Continuous batching, introduced for LLM serving by the Orca system as iteration-level scheduling and used by vLLM and other engines (TensorRT-LLM calls it in-flight batching), re-forms the batch between decode steps, so a slot is refilled the moment a sequence ends. Throughput and slot utilisation improve substantially, and new requests start almost immediately instead of waiting for a whole batch to drain. What limits it in practice is KV-cache memory: the scheduler must decide whether another sequence fits, and may queue or preempt when it does not. Long prefills can also stall the decodes of running sequences, which is why engines add chunked prefill (splitting a long prompt across iterations).

Here is the mechanism reduced to a few lines. The snippet counts decode steps for the same 400 requests (seeded, illustrative length distribution) under both policies with 8 slots:

import random, statistics

rng = random.Random(7)
N, SLOTS = 400, 8
out_len = [min(300, max(5, int(rng.lognormvariate(4.2, 0.9)))) for _ in range(N)]   # output tokens per request

def static_batching(lens):
    """Fill 8 slots, run until the LONGEST sequence is done, then start the next batch.
    Every result in a batch is returned when the whole batch ends; short sequences idle their slot."""
    step, done_at = 0, []
    for i in range(0, len(lens), SLOTS):
        batch = lens[i:i + SLOTS]
        step += max(batch)
        done_at += [step] * len(batch)
    return step, done_at

def continuous_batching(lens):
    """Iteration-level scheduling: after EVERY decode step, finished sequences leave and waiting ones join."""
    waiting, running, step, done_at = list(lens), [], 0, []
    while waiting or running:
        while waiting and len(running) < SLOTS:
            running.append(waiting.pop(0))
        step += 1
        running = [r - 1 for r in running]
        done_at += [step] * sum(1 for r in running if r == 0)
        running = [r for r in running if r > 0]
    return step, done_at

total = sum(out_len)
s_steps, s_done = static_batching(out_len)
c_steps, c_done = continuous_batching(out_len)
print(f"requests={N} slots={SLOTS} total output tokens={total} longest={max(out_len)} mean={total/N:.1f}")
for name, steps, done in (("static    ", s_steps, s_done), ("continuous", c_steps, c_done)):
    print(f"{name}: {steps:5d} decode steps, slot utilisation {100*total/(steps*SLOTS):5.1f}%, "
          f"mean completion step {statistics.mean(done):7.1f}")
print(f"same work finished in {s_steps/c_steps:.2f}x fewer decode steps with continuous batching")

# Output:
# requests=400 slots=8 total output tokens=33816 longest=300 mean=84.5
# static    : 10336 decode steps, slot utilisation  40.9%, mean completion step  5160.1
# continuous:  4303 decode steps, slot utilisation  98.2%, mean completion step  2115.1
# same work finished in 2.40x fewer decode steps with continuous batching

Slot utilisation is the fraction of slot-steps that produced a useful token. The 2.40x figure is specific to this synthetic length distribution (a long tail up to 300 tokens); with uniform output lengths the two policies converge, which is worth saying in an interview: continuous batching pays off in proportion to length variance.

Deep Dive: Latency vs Throughput and the GPU Utilisation Trap

latency of a request  =  queue wait (for batch to fill / GPU to free)
                       +  service time (grows with batch size)
                       +  (for LLMs) time per output token x tokens

  small batch, no wait :  low latency, GPU mostly idle, high cost per request
  large batch, long wait:  high throughput, low cost, latency up for EVERYONE in the batch

Tail latency (p99) is set by the worst case: a request that arrives just after a batch launched
waits for that whole batch, then possibly waits max_wait, then runs in a full (slower) batch.

Batching converts idle GPU time into cheaper requests, but the bill is paid in latency, and p99 suffers more than p50. Tune against a stated SLO: for example "p99 under 200 ms" fixes how much wait and batch size you can afford, and the best configuration is the one that maximises throughput inside that budget. Note also that the utilization figure reported by nvidia-smi generally measures the fraction of time at least one kernel was running, not how efficiently the compute units were used, so a GPU can show high utilisation while doing tiny, memory-bound work. Use throughput per GPU at the target latency (tokens/s or requests/s) as the metric that matters. See Capacity Planning & SLOs and Autoscaling for Inference Workloads for how this feeds capacity decisions.

Key Pointers

• Batching is efficient because small-batch inference is memory-bandwidth bound: weights are read once per pass no matter how many rows share the pass.
• Static batching waits for a fixed size and has unbounded queueing delay at low traffic; dynamic batching launches on "batch full OR max wait elapsed", bounding the added delay.
• The two knobs are max batch size (limited by memory and step time) and max wait (limited by your latency SLO); tune them against a p99 target, not the mean.
• Padding to the longest sequence wastes compute; bucketing by length, packing, or ragged kernels reduce it, at the cost of more queues or more complexity.
• For LLM generation, batches with different output lengths suffer head-of-line blocking; continuous (iteration-level) batching refills slots after every decode step.
• KV-cache memory (roughly 2 x layers x kv_heads x head_dim x bytes per token, per sequence) caps the achievable batch size on long contexts.
• Under heavy load batching costs no extra wait because requests accumulate while the GPU is busy; the wait timer matters mostly at low and moderate load.
• GPU "utilisation" from nvidia-smi is not efficiency; judge by throughput at the target latency.

How It Works

1. Requests enter a queue, each stamped with its arrival time and (for text) its token count.
2. The batcher checks a launch rule: enough requests to fill max batch size, or the oldest request has waited max wait, and the GPU is free to run something.
3. It picks up to max-batch requests (optionally grouped by similar length to limit padding) and assembles them into one padded or packed tensor.
4. The model runs one forward pass; for LLMs a continuous-batching scheduler instead runs one iteration at a time, running prefill for new sequences and one decode step for active ones.
5. Outputs are split per request and returned to their callers; in continuous batching, sequences that emitted an end token leave immediately and free their KV-cache blocks.
6. The scheduler loops, admitting waiting requests into any freed capacity, while metrics (queue depth, batch size, wait time, p99) feed tuning and autoscaling.

Try It Yourself

A seeded discrete-event simulation of one GPU serving Poisson traffic. Batch service time is 8 ms + 0.02 ms x padded_tokens, an assumed cost model (fixed pass overhead plus per-padded-token compute), not a benchmark of any real hardware. It compares no batching, static batching (wait for exactly 8) and dynamic batching (max 8, wait up to 10 ms) at three load levels; latency includes queue wait.

import random, statistics

OVERHEAD_MS = 8.0      # fixed cost per forward pass (kernel launches, weight reads, scheduling)
PER_TOKEN_MS = 0.02    # cost per padded token in the batch (illustrative, not a measurement)

def batch_time(batch_lens):
    padded = len(batch_lens) * max(batch_lens)          # every row is padded to the longest
    return OVERHEAD_MS + PER_TOKEN_MS * padded

def make_traffic(rate_per_s, n, seed):
    rng = random.Random(seed)
    t, arrivals, lens = 0.0, [], []
    for _ in range(n):
        t += rng.expovariate(rate_per_s) * 1000.0        # ms, Poisson arrivals
        arrivals.append(t)
        lens.append(min(200, max(8, int(rng.lognormvariate(4.0, 0.6)))))   # input tokens
    return arrivals, lens

def simulate(arrivals, lens, max_batch, max_wait_ms):
    """Single server. A batch launches when it is full, or when the oldest request has
    waited max_wait_ms (and the server is free). max_wait_ms=inf means 'wait until full'."""
    n, i, free_at = len(arrivals), 0, 0.0
    queue, latencies, real_tokens, padded_tokens, busy, last_finish, sizes = [], [], 0, 0, 0.0, 0.0, []
    while i < n or queue:
        if not queue:
            queue.append(i); i += 1
        while i < n and arrivals[i] <= free_at:           # absorb arrivals while the server was busy
            queue.append(i); i += 1
        if len(queue) >= max_batch:
            launch = max(free_at, arrivals[queue[max_batch - 1]])
        else:
            deadline = max(free_at, arrivals[queue[0]] + max_wait_ms)
            launch = None
            while i < n and arrivals[i] <= deadline:
                queue.append(i); i += 1
                if len(queue) >= max_batch:
                    launch = max(free_at, arrivals[queue[-1]]); break
            if launch is None:
                launch = deadline if deadline != float("inf") else max(free_at, arrivals[queue[-1]])
        batch, queue = queue[:max_batch], queue[max_batch:]
        bl = [lens[k] for k in batch]
        svc = batch_time(bl)
        finish = launch + svc
        for k in batch: latencies.append(finish - arrivals[k])
        real_tokens += sum(bl); padded_tokens += len(bl) * max(bl)
        busy += svc; sizes.append(len(batch)); free_at = finish; last_finish = finish
    latencies.sort()
    pct = lambda p: latencies[min(len(latencies) - 1, int(p * len(latencies)))]
    return dict(thr=n / (last_finish / 1000.0), p50=pct(0.50), p99=pct(0.99),
                avg_b=statistics.mean(sizes), pad=1 - real_tokens / padded_tokens,
                util=busy / last_finish)

POLICIES = [("no batching     (max 1)         ", 1, 0.0),
            ("static          (wait for 8)    ", 8, float("inf")),
            ("dynamic         (max 8, wait 10)", 8, 10.0)]
print("policy                           load  thr/s  p50ms   p99ms  avg_b  pad%  gpu_busy%")
for rate in (20, 80, 160):
    arr, ln = make_traffic(rate, 20000, seed=42)
    for name, mb, mw in POLICIES:
        r = simulate(arr, ln, mb, mw)
        print(f"{name}  {rate:4d}  {r['thr']:5.1f} {r['p50']:7.1f} {r['p99']:8.1f}  {r['avg_b']:4.1f}  {100*r['pad']:4.1f}  {100*r['util']:6.1f}")

# Output:
# policy                           load  thr/s  p50ms   p99ms  avg_b  pad%  gpu_busy%
# no batching     (max 1)             20   20.1     9.3     22.8   1.0   0.0    18.7
# static          (wait for 8)        20   20.1   179.2    609.9   8.0  51.0     7.3
# dynamic         (max 8, wait 10)    20   20.1    19.2     26.0   1.2  10.3    16.3
# no batching     (max 1)             80   80.4    17.6     83.5   1.0   0.0    74.7
# static          (wait for 8)        80   80.4    66.4    175.3   8.0  51.0    29.1
# dynamic         (max 8, wait 10)    80   80.4    19.9     33.4   1.8  25.7    49.2
# no batching     (max 1)            160  107.7 32216.0  60913.5   1.0   0.0   100.0
# static          (wait for 8)       160  160.8    49.6    103.9   8.0  51.0    58.2
# dynamic         (max 8, wait 10)   160  160.9    22.6     52.6   2.9  36.8    77.2

How to read it: at 20 req/s unbatched serving is already best (nothing to amortise, the GPU is mostly idle), static batching costs about 180 ms at p50 just waiting for company, and dynamic batching sits about 10 ms above unbatched (the price of the max_wait timer). At 80 req/s the unbatched server is at ~75% busy and its p99 (83.5 ms) starts to climb, while dynamic batching cuts p99 to 33.4 ms. At 160 req/s the unbatched server cannot keep up (its capacity here is about 107 req/s, so its queue grows without bound and the reported latency is meaningless), while both batching policies sustain the load, with dynamic batching roughly halving static's latency. The throughput column equals the arrival rate whenever the system is stable; the interesting columns are p50, p99 and gpu_busy. All numbers come from the assumed cost model, so treat the shape as the lesson, not the values.

Interview Q&A

Q: Why does batching increase throughput on a GPU?

A: For small batches the GPU is limited by reading weights from memory and by kernel-launch overhead, not by arithmetic, so a pass over 8 rows takes barely longer than a pass over 1. Cost per request is roughly step time divided by batch size, which falls fast until compute or memory capacity becomes the limit. The trade-off is that every request in the batch waits for the batch, so latency rises.

Q: What is the difference between static and dynamic batching?

A: Static batching waits for a fixed number of requests, so latency depends on traffic and can be unbounded at low load. Dynamic batching launches when the batch is full or the oldest request has waited a maximum time, so added delay is bounded and batches naturally grow under load. You now have two parameters, max batch size and max wait, to tune against your latency SLO.

Q: How do you choose max wait?

A: Work backwards from the latency budget: max wait plus the service time of a full batch plus network and pre/post-processing must fit inside the p99 target. If the target is 200 ms and a full batch takes 60 ms, you cannot spend more than a small fraction of the remaining budget waiting. Then measure: increase max wait until batch size stops rising or p99 approaches the limit. It matters mainly at low load; at high load the queue fills on its own.

Q: What is padding waste and how do you reduce it?

A: Rows in a tensor must share a shape, so shorter sequences are padded to the longest in the batch and the GPU computes on tokens it discards; with lengths of 10, 24, 8 and 12 in one batch, 43.75% of slots are padding. Reduce it by grouping requests of similar length, capping input length, or using packed or ragged kernels. Bucketing costs you more queues that each fill more slowly.

Q: What is continuous batching and why is it better for LLMs?

A: Generation runs for a variable number of decode steps per request, so in static batching short requests idle until the longest finishes and new requests wait for the whole batch. Continuous (iteration-level) batching re-forms the batch after every decode step, letting finished sequences leave and waiting ones join. In the simulation above, the same 400 requests needed 4,303 steps instead of 10,336 with 8 slots, though that ratio depends on how skewed output lengths are.

Q: Does bigger batch always mean better GPU efficiency?

A: No. Past a point the step becomes compute-bound or memory-limited: step time grows roughly linearly, so throughput flattens while latency keeps rising. Padding waste also grows with batch size, and KV-cache memory can cap the batch long before compute does. The right batch size is the largest one that still meets the latency SLO.

Follow-up: Your p50 looks fine but p99 doubled after you enabled dynamic batching. What do you check?

A: Look at max wait relative to typical inter-arrival time, and at batch size distribution: a request arriving just after a launch waits for the running batch and then for its own timer, which hits the tail, not the median. Check whether variable lengths cause padded, slow batches, and whether a few long requests dominate step time. Fixes are a lower max wait, length bucketing, or a smaller max batch; also confirm queue depth is not rising, which would mean you are simply over capacity.

Follow-up: A team says nvidia-smi shows 95% GPU utilisation, so batching cannot help. Are they right?

A: Not necessarily. That metric generally records the share of time in which at least one kernel was executing, so a GPU running many tiny memory-bound kernels can read 95% while its compute units are mostly idle. Measure tokens or requests per second at the target latency, and try a larger batch or continuous batching: if throughput rises without violating p99, there was headroom.

Follow-up: How does batching interact with autoscaling?

A: Batching raises per-GPU capacity as load increases, so CPU-style or request-rate triggers can under-react: throughput per replica is not constant. Scale on queue depth, wait time, or a latency SLO signal instead, and remember new GPU replicas take minutes to load weights, so you keep headroom rather than reacting only to saturation. Also, adding replicas splits traffic, which shrinks each replica's batches and can lower efficiency.

System Design Tie-in

Batching appears in every inference design: an LLM API gateway routes to vLLM or TensorRT-LLM workers that do continuous batching internally, a recommender scores candidate lists as one batch of items per request, and embedding pipelines batch documents offline for throughput. The estimation an interviewer wants is an ILLUSTRATIVE one, with stated assumptions: suppose peak 200 requests/s, each needing 500 output tokens, so 100,000 output tokens/s of demand; if one GPU sustains, say, 2,500 tokens/s at your latency target with batching (an assumption you must replace with your own measurement), you need about 40 GPUs plus headroom, and the same GPU without batching might sustain a small fraction of that, which is why batching dominates cost. State the SLO first (for example p99 time to first token and time per output token), because it caps batch size and wait. In the design also mention queue-depth-based autoscaling, admission control or load shedding when queues exceed a bound (see Rate Limiting & Throttling), and separating latency-sensitive interactive traffic from throughput-oriented offline traffic, which can use much larger batches on a different pool. Related: Model Serving Architectures, Model Serving Frameworks and Cost Optimization for AI Systems.

Trade-offs

The decision is how much per-request latency to trade for GPU efficiency. Larger batches and longer waits lower cost per request and raise throughput, but they add queueing delay for everyone, worsen the tail, waste compute on padding when lengths vary, and consume KV-cache memory. Dynamic batching removes the worst failure of static batching (unbounded waits at low traffic) at the price of two parameters that must be re-tuned when traffic shape, model, or hardware changes, and continuous batching adds scheduler complexity, memory-management concerns (preemption, KV-cache eviction) and interference between prefill and decode. Batching is the wrong choice, or should be nearly disabled, when traffic is low and latency-critical (unbatched requests are already fast and the GPU is idle anyway), when a single request already saturates the GPU (very large inputs), or when a hard real-time deadline leaves no room for any wait. It also does not help if requests cannot share a tensor (very different shapes, different models) without heavy padding, in which case route them to separate pools.