Streaming means sending an LLM's output to the client incrementally, token by token or in small groups, over a connection that stays open, instead of waiting for the whole completion and returning it as one response. Interviewers probe it because it is the one place where a plain request/response mental model breaks: latency has two numbers instead of one, failures can happen after you have already sent a 200 OK, proxies silently undo the whole idea by buffering, and a disconnected client can leave a GPU generating tokens nobody will read. For a Backend or AI engineer it decides the transport (see API Design), what the gateway has to forward, how you bill, and how the mobile client behaves on a flaky network.
Streaming means an app shows you an answer piece by piece as it's being created, instead of making you stare at a blank screen until the whole thing is finished and then dumping it all at once. It's the difference between watching words appear on screen as an AI "types" its reply and waiting in silence for thirty seconds before the full paragraph pops up.
It's like ordering food at a busy diner counter versus a drive-through with one single hand-off window. At the counter, the cook hands you each item as it's ready — fries first, then the burger, then the drink — so you start eating right away. At the drive-through, you wait at the window until the entire bag is packed before you get anything at all, even though the total wait might be the same.
Generation has two phases (see llm-inference-fundamentals.html):
prefill : process the whole prompt in parallel -> produces the FIRST token
decode : one token per step, each depends on the previous one (sequential)
time -->
|-- queue --|-- prefill --|t1|t2|t3|t4| ............ |t500|
^ ^ ^
request sent first token (TTFT) last token
Non-streaming: user sees nothing until the last token (TTFT + N / rate)
Streaming : user sees text from the first token on (TTFT)# ILLUSTRATIVE assumptions: TTFT 0.4 s, decode 40 tok/s, 500 output tokens, ~60 bytes per SSE frame
ttft, rate, n_tokens, frame_bytes = 0.4, 40, 500, 60
total = ttft + n_tokens / rate
print("non-streaming: first pixel at %.1f s" % total)
print("streaming : first pixel at %.1f s (finished at %.1f s)" % (ttft, total))
# Proxy that buffers 4096 bytes before forwarding (e.g. response buffering left on)
buf_bytes = 4096
tokens_to_fill = -(-buf_bytes // frame_bytes) # ceiling division
first_flush = ttft + tokens_to_fill / rate
print("buffering proxy: needs %d tokens to fill %d B -> first pixel at %.1f s, then bursts of %d tokens every %.1f s"
% (tokens_to_fill, buf_bytes, first_flush, tokens_to_fill, tokens_to_fill / rate))
# Real output from python3:
# non-streaming: first pixel at 12.9 s
# streaming : first pixel at 0.4 s (finished at 12.9 s)
# buffering proxy: needs 69 tokens to fill 4096 B -> first pixel at 2.1 s, then bursts of 69 tokens every 1.7 sDecode is sequential by construction, so total time is roughly TTFT plus output tokens divided by decode rate, and you cannot shorten it much by being clever in the web tier. What you can change is when the user first sees something. Time to first token (TTFT) is dominated by queueing, prompt length (prefill cost) and any retrieval or tool calls before generation; inter-token latency (ITL) is the steady decode speed. Streaming leaves total time unchanged, and total work is essentially unchanged too, but it converts a 13-second blank wait into a 0.4-second wait followed by text arriving at roughly reading speed. The numbers above are ILLUSTRATIVE assumptions, not measurements. Streaming also does not help when the consumer is a program that needs the complete answer (a JSON parser, a batch job), where it only adds framing overhead.
SSE WebSocket plain chunked HTTP
direction server -> client full duplex server -> client
protocol HTTP, text/event-stream HTTP Upgrade, then WS HTTP/1.1 Transfer-Encoding: chunked
(HTTP/2+ has framing natively)
framing "field: value" lines, binary/text messages, none; you invent it (NDJSON, ...)
blank line ends event you define the schema
auto-reconnect yes (EventSource) with no, you build it no, you build it
Last-Event-ID
proxies / LBs ordinary HTTP, usually need Upgrade support, ordinary HTTP
fine idle timeouts, sticky
request body EventSource: GET only any any (POST works)
best for model -> client text voice, live tool loops, simple internal service
interactive sessions to service streams
SSE frame on the wire:
id: 7
event: token
data: {"t":"hello"}
(blank line)Most text streaming is one-directional after the initial prompt, so SSE is the default: it is plain HTTP that load balancers, auth middleware and observability already understand, with a small event grammar and, in the browser's EventSource, automatic reconnect that sends Last-Event-ID. Its limits are practical: EventSource only issues GET without custom headers, so many LLM clients use fetch with a POST body and parse the text/event-stream body themselves (which loses the built-in reconnect); on HTTP/1.1 browsers cap concurrent connections per origin (commonly around six), which HTTP/2 multiplexing removes; and it is text only. WebSockets earn their complexity when the client must interrupt or send audio while the model speaks, or when you already need bidirectional messaging; the cost is a stateful connection that load balancers must keep pinned, custom reconnect and heartbeat logic, and less reuse of HTTP tooling. Details of the protocol are in WebSockets; framing choices also connect to the HTTP versions in HTTP/1.1 vs HTTP/2 vs HTTP/3.
engine -> API server -> gateway -> reverse proxy / LB -> CDN? -> client
each hop may BUFFER, COMPRESS, or IDLE-TIMEOUT the stream
Symptoms of a buffering hop:
- works on localhost, arrives in one lump in production
- text appears in bursts every N seconds instead of smoothly
Typical causes and knobs (nginx as the example; other proxies have equivalents):
proxy_buffering off; # or response header X-Accel-Buffering: no
gzip on ... # compression buffers until a block fills: exclude text/event-stream
proxy_read_timeout 300s; # idle timeout between chunks, not total time
send ": keepalive\n\n" every ~15 s so idle-connection reapers do not kill a slow streamThe stream is only as incremental as its least cooperative hop. A proxy that buffers a response until 4 KB accumulates turns per-token delivery into bursts (in the illustrative arithmetic above, about 69 small frames per burst), and gzip or a CDN that aggregates chunks does the same. Serverless platforms and some managed gateways have response-size or duration limits or do not stream at all, so verify with a real end-to-end test, not a unit test. Idle timeouts differ from total timeouts: a stream that pauses during a tool call or a long prefill can be reaped unless you send comment frames, which is why heartbeat lines are standard.
Backpressure is the second half. If the client or network reads slower than the model produces, bytes queue in socket buffers and then in your process, and an unbounded queue per connection is an out-of-memory bug waiting for a slow mobile client at 10,000 concurrent streams. The engine is generally shared by many requests in a batch (continuous batching), so you cannot slow the GPU for one reader; the server instead uses bounded per-connection buffers (in async code, awaiting drain() or a bounded queue), coalesces several pending tokens into one frame when the reader lags (tokens must not be dropped, but they can be merged), and disconnects clients that stay too far behind. See Reverse Proxy and Load Balancing for the hops involved.
HTTP status line is sent BEFORE generation finishes:
Non-streaming failure : 500 -> client retries the whole request, nothing was shown
Streaming failure : 200 OK already sent, 5 tokens already rendered, then the connection dies
-> status codes can no longer signal the error; use an in-band
`event: error` frame, or detect the abrupt close / missing `done` frame
Three recovery choices after a partial answer:
1. discard and restart simple; user sees text vanish; you pay for the tokens again
2. resume from a buffer server keeps emitted events per stream id; client sends Last-Event-ID
3. continue from the prefix send prompt + partial text as a new call; a NEW generation, may not match
Cancellation: client disconnects -> server must notice -> engine must ABORT the request
otherwise the GPU decodes up to max_tokens for nobodyBecause the 200 status is already on the wire, a stream needs its own protocol for terminal states: a final done event carrying usage, and an explicit error event; a connection that ends without done means truncated. Resuming from a server-side buffer is cheap and deterministic (the Try It Yourself demo replays token 4, which was generated but only partly delivered), but it requires stickiness or a shared store keyed by stream id, and it must expire. Restarting generation with the same prompt is not guaranteed to reproduce the earlier text unless sampling is deterministic, and even then batch composition can change numerics, so a naive retry can contradict what the user already read. Retrying a non-idempotent side effect (a tool call that sent an email) needs idempotency keys; see Idempotency and Exactly-Once Semantics and Timeouts, Retries and Backoff.
Cancellation is where streaming touches cost. A closed socket is only visible to the code that writes to it, so the API server has to translate a failed write or a disconnect signal into an abort call on the inference engine (engines such as vLLM generally expose a request-abort path for this; check your framework), which frees the sequence's KV-cache blocks and its batch slot immediately. Without it, the last case in the demo below shows generation running to max_tokens with no reader. Note the in-flight lag too: one token more was generated than the client displayed, because tokens are in flight when the disconnect happens. For accounting, count output tokens on the server as they are generated and put the total in the final frame; on cancel or failure there is no final frame, so the gateway has to record usage itself, and providers generally bill for tokens generated even if the client never received them (check current provider terms).
Stream of raw JSON text: {"city": "Par is", "temp": 21, "un
Complete only at the end -> naive JSON.parse fails on every prefix.
Options
1. buffer until closed, then parse (no benefit of streaming for the data)
2. incremental / partial-JSON parser (tolerates prefixes; values may still change)
3. stream fields as separate events (server parses, emits {"field":"city","value":"Paris"})
4. tool calls: arguments arrive as string fragments; execute only after the call is complete
Mobile client concerns
radio/network switch (Wi-Fi <-> LTE) drops the TCP connection -> must resume, not restart
app backgrounded -> OS suspends sockets -> reconnect on foreground with last event id
render throttling: coalesce tokens and update the UI at frame rate (~16-60 ms), not per token
battery: keep-alive intervals and long-lived radios cost power; stop the stream when the screen leavesStructured output is awkward because JSON is only valid when closed. If a downstream component needs an object, either wait for completion or use a tolerant incremental parser and treat partial values as provisional (a number can grow from 2 to 21, a string can still change); never act on a partial tool call. Constrained decoding, where the engine masks invalid tokens so output always matches a schema, helps validity at completion but does not remove the prefix problem. On mobile, the network is the least reliable hop: connections die on handoff, backgrounding suspends them, and UI updates per token can cost more than the network. A good Android or iOS client keeps the last event id, reconnects with backoff, appends resumed tokens to the same message, batches rendering, and cancels the request when the user leaves the screen so the server can free the GPU.
Last-Event-ID. Use WebSockets when the client must send while the model speaks (voice, interruptions, live tool loops).done event with usage; treat a missing done as truncation and choose restart, buffered resume, or prefix continuation deliberately.max_tokens for no reader.Content-Type: text/event-stream and disables response buffering for this route.id and writes it to the socket.Last-Event-ID; the server replays from a per-stream buffer or the client restarts, and a missing done event marks the answer incomplete.done with usage for accounting; on client disconnect it aborts the request in the engine, frees KV-cache blocks and records tokens generated so far.The program below simulates the model with a generator (no sleeping, no network), frames tokens as SSE, splits the byte stream into arbitrary 17-character chunks, parses it incrementally on the client, kills the server mid-stream, resumes with the last id, and then compares cancellation with a server that never notices the disconnect.
# Python 3 stdlib: SSE token stream, incremental client parser, mid-stream failure, resume, cancellation
import json
WORDS = "Streaming lets the user read the answer while the model is still generating it".split()
class ConnectionReset(Exception):
pass
class Server:
"""Simulates a model server. `generated` counts tokens the (pretend) GPU produced."""
def __init__(self):
self.generated = 0
self.buffers = {} # stream_id -> list of emitted events (resume buffer)
def sse(self, stream_id, start=0, max_tokens=12, fail_at=None):
"""Generator of SSE frames (str). Closing it early runs `finally` = cancellation."""
buf = self.buffers.setdefault(stream_id, [])
try:
yield ": keepalive\n\n" # comment frame, ignored by clients
for i in range(start, max_tokens):
if i < len(buf): # replay from resume buffer
tok = buf[i]
else:
if fail_at is not None and i == fail_at:
raise ConnectionReset("upstream died at token %d" % i)
tok = WORDS[i % len(WORDS)] # 'decode' one token
self.generated += 1
buf.append(tok)
yield "id: %d\nevent: token\ndata: %s\n\n" % (i, json.dumps({"t": tok}))
yield 'event: done\ndata: {"usage": {"output_tokens": %d}}\n\n' % len(buf)
finally:
print(" [server] stream %s closed, GPU tokens so far: %d" % (stream_id, self.generated))
def chunk_bytes(frames, size=17):
"""Network does not respect frame boundaries: re-slice into arbitrary chunks."""
pending = ""
for frame in frames: # lazy: pulls one frame at a time
pending += frame
while len(pending) >= size:
yield pending[:size]
pending = pending[size:]
if pending:
yield pending
class SSEParser:
def __init__(self):
self.buf, self.last_id = "", None
def feed(self, chunk):
self.buf += chunk
events = []
while "\n\n" in self.buf: # a blank line ends one event
raw, self.buf = self.buf.split("\n\n", 1)
ev = {"event": "message", "data": []}
for line in raw.split("\n"):
if line.startswith(":") or not line:
continue # comment / keepalive
field, _, value = line.partition(":")
value = value.lstrip(" ")
if field == "data": ev["data"].append(value)
elif field == "event": ev["event"] = value
elif field == "id": self.last_id = int(value); ev["id"] = self.last_id
if ev["data"]:
ev["data"] = "\n".join(ev["data"]); events.append(ev)
return events
def consume(server, stream_id, start=0, fail_at=None, cancel_after=None):
parser, text, seen = SSEParser(), [], 0
frames = server.sse(stream_id, start=start, fail_at=fail_at)
try:
for chunk in chunk_bytes(frames):
for ev in parser.feed(chunk):
if ev["event"] == "token":
text.append(json.loads(ev["data"])["t"]); seen += 1
if cancel_after and seen == cancel_after:
frames.close() # client disconnects
return " ".join(text), "cancelled", parser.last_id
elif ev["event"] == "done":
return " ".join(text), "done " + ev["data"], parser.last_id
except ConnectionReset as e:
return " ".join(text), "error: %s" % e, parser.last_id
return " ".join(text), "eof", parser.last_id
print("1) happy path, frames split into 17-char chunks")
s = Server()
print(" ", consume(s, "a"))
print("2) mid-stream failure at token 5, then resume with Last-Event-ID")
s = Server()
partial, status, last_id = consume(s, "b", fail_at=5)
print(" partial:", repr(partial), "|", status, "| last_id =", last_id)
rest, status2, _ = consume(s, "b", start=last_id + 1)
print(" resumed:", repr(rest), "|", status2)
print(" full text:", partial + " " + rest, "| GPU tokens generated:", s.generated)
print("3) cancellation after 3 tokens (client closes the stream)")
s = Server()
print(" ", consume(s, "c", cancel_after=3))
print(" GPU tokens generated:", s.generated, "of max 12")
print("4) same disconnect but server never notices (no cancel propagation)")
s = Server()
for _ in s.sse("d"): pass # generation runs to max_tokens for nobody
print(" GPU tokens generated:", s.generated, "of max 12 (nobody is reading)")
# Real output from python3:
# 1) happy path, frames split into 17-char chunks
# [server] stream a closed, GPU tokens so far: 12
# ('Streaming lets the user read the answer while the model is still', 'done {"usage": {"output_tokens": 12}}', 11)
# 2) mid-stream failure at token 5, then resume with Last-Event-ID
# [server] stream b closed, GPU tokens so far: 5
# partial: 'Streaming lets the user' | error: upstream died at token 5 | last_id = 3
# [server] stream b closed, GPU tokens so far: 12
# resumed: 'read the answer while the model is still' | done {"usage": {"output_tokens": 12}}
# full text: Streaming lets the user read the answer while the model is still | GPU tokens generated: 12
# 3) cancellation after 3 tokens (client closes the stream)
# [server] stream c closed, GPU tokens so far: 4
# ('Streaming lets the', 'cancelled', 2)
# GPU tokens generated: 4 of max 12
# 4) same disconnect but server never notices (no cancel propagation)
# [server] stream d closed, GPU tokens so far: 12
# GPU tokens generated: 12 of max 12 (nobody is reading)
# Reading the output:
# - case 2: last_id = 3, not 4. Token 4's frame was cut by a chunk boundary when the failure hit,
# so the client never saw a complete event and resumes at 4; the server replays it from its buffer.
# - case 3: 4 tokens generated for 3 displayed: one token was in flight when the client closed.
# - case 4: 12 tokens of GPU work for a reader who left; that is the waste cancellation prevents.
# Deployment config (not run here; nginx, output not shown):
# location /v1/chat/stream {
# proxy_pass http://inference;
# proxy_http_version 1.1;
# proxy_buffering off;
# proxy_read_timeout 300s;
# gzip off;
# }Q: Why does streaming improve UX if total generation time is the same?
A: Perceived latency is time until the user sees progress, and streaming moves that from TTFT plus full decode to TTFT alone. In an illustrative case of a 0.4 s TTFT, 40 tokens/s and 500 tokens, the wait to first text drops from about 13 s to 0.4 s while completion still takes about 13 s. The trade-off is added framing, connection management and error-handling complexity, and no benefit for machine consumers that need the whole answer.
Q: SSE or WebSockets for a chat product?
A: SSE for the text path: it is one-directional after the prompt, rides ordinary HTTP through existing auth, load balancers and logging, and has a defined reconnect mechanism. I would choose WebSockets when the client needs to send during generation, such as voice input, barge-in or interactive tool sessions. The cost of WebSockets is stateful connections, sticky routing, custom heartbeats and reconnection, so I would not pay it for a read-only token stream.
Q: The stream works locally but arrives in one lump in production. What do you check?
A: A buffering or compressing hop: reverse-proxy response buffering, gzip on the event-stream content type, a CDN or API gateway that aggregates, or a serverless platform that does not stream. I would curl the endpoint through each layer with no client buffering, then set proxy_buffering off or X-Accel-Buffering: no and exclude the route from compression. Also check idle timeouts, since a stream that pauses during prefill or a tool call gets cut unless heartbeats are sent.
Q: What happens if the connection dies after 200 tokens?
A: The status code was already 200, so the client must detect a missing terminal done event or an abrupt close. Then I pick a recovery: resume from a server-side buffer with Last-Event-ID (cheap and consistent, needs a store or stickiness and expiry), restart (simple, wastes cost and may contradict text already shown), or continue from the partial text as a new prompt (fewer tokens, but a different generation). Any tool side effects need idempotency keys so retries do not repeat them.
Q: How do you stop paying for GPU work when a user closes the tab?
A: Propagate the disconnect to the inference engine as an abort. The API server notices the failed write or disconnect signal and cancels the request, which releases its KV-cache blocks and batch slot; frameworks such as vLLM generally expose an abort path, but verify in yours. Without it, a 4,000-token limit means thousands of wasted decode steps per abandoned request, which at scale is a real fraction of GPU capacity.
Q: How do you count tokens for billing on a stream?
A: Count output tokens server-side as they are produced and send the total in the final event; input tokens are known from the tokenizer at prefill. The gap is failures and cancellations, which never send a final event, so the gateway records usage on close regardless of cause, and the policy on billing for undelivered tokens must be explicit (providers generally bill for what was generated; check terms). Client-reported counts should never be trusted for billing.
Q: How would you handle a slow client that reads far slower than tokens arrive?
A: Use a bounded per-connection buffer and let the socket's flow control apply backpressure to that connection only. Because the engine's batch is shared, it cannot pause for one reader, so the server coalesces pending tokens into larger frames while the client lags and disconnects it past a limit. An unbounded queue at 10,000 concurrent streams turns one slow mobile network into an out-of-memory incident.
Follow-up: EventSource cannot POST or set headers. How do real LLM clients cope?
A: They use fetch with a POST body and read the response body as a stream, parsing the SSE grammar themselves, or a library that does so. The price is that automatic reconnect and Last-Event-ID handling are gone, so the client must implement reconnection, backoff and resume explicitly; also keep prompts out of URLs, since query strings are logged.
Follow-up: How do you stream structured output such as JSON safely?
A: A prefix of JSON is not valid JSON, so either buffer to completion, use a tolerant incremental parser and treat partial values as provisional, or have the server parse and emit typed field events. Tool-call arguments arrive as string fragments and must not be executed until the call is complete and validated. Constrained decoding improves final validity but does not make prefixes parseable.
Follow-up: How would you scale a stream-holding tier to 100,000 concurrent connections?
A: Each stream is a long-lived connection holding memory (socket buffers, the resume buffer, parser state) but little CPU, so use an event-driven server, keep per-stream state small, and set file-descriptor and connection limits deliberately. Scale on concurrent connections rather than requests per second, drain gracefully on deploys (finish or hand off streams, do not kill them mid-answer), and keep the resume buffer in shared memory or a cache if reconnects can land on a different instance. The GPU tier scales separately on tokens per second, so the two need different autoscaling signals.
In a chat or copilot design, streaming is the first thing to state after the diagram: client to edge to gateway to inference, SSE over HTTP/2, buffering disabled on every hop, heartbeats every 15 seconds. Make capacity estimates in the right unit, which is concurrent streams: ILLUSTRATIVELY, 50,000 active users each holding a stream for an average of 15 seconds with one message every 60 seconds gives roughly 50,000 x 15 / 60, about 12,500 concurrent streams, each a small memory footprint but a long-lived socket that constrains connection limits and rolling deploys. Bandwidth is small (a few KB per second per stream), so the constraint is connection count and inference throughput, not the network. Then address the failure story: terminal done and error frames, resume buffers with a TTL, idempotency for tool calls, abort on disconnect, usage recorded at the gateway, and mobile reconnect. Mention the guardrail interaction as well: streaming forces moderation to operate on partial text or on delayed windows, so a filter that must see the whole answer conflicts with low TTFT (see Guardrails, Content Moderation and Safety). The gateway must proxy streams without buffering them (see LLM Gateway and Multi-Provider Routing).
You are buying perceived responsiveness with operational complexity: long-lived connections that strain proxies and deploys, a protocol for partial failure and resume, cancellation plumbing into the engine, client parsers, and moderation that has to work on incomplete text. SSE trades bidirectionality and binary support for HTTP compatibility; WebSockets do the reverse; plain chunked responses are simplest but give you no reconnect or framing. Resume buffers trade memory and stickiness for continuity; restarting trades tokens and consistency for simplicity. Streaming is the wrong choice for short completions where TTFT is already tiny, for machine-to-machine calls that need a complete validated object, for batch and offline jobs where a queue is better (see Synchronous vs Asynchronous Processing), and for environments whose proxy or serverless platform cannot pass chunks through without buffering. Measure TTFT, inter-token latency and completed-stream rate rather than assuming that streaming alone fixes a slow model.