A very knowledgeable friend on a phone call who speaks word by word as they think. Every time you ask something, you quickly remind them of the conversation so far (they don't remember on their own), and the longer the chat, the longer that reminder gets.
Problem Statement
Design a chat assistant like ChatGPT or Claude: users hold multi-turn conversations with a large language model, see the answer stream in token by token, and come back later to their conversation history. The unusual parts compared with a normal web app are that each request is expensive GPU work lasting seconds, the response is a stream, and the "context" sent to the model grows with every turn.
Functional Requirements
Start a conversation, send a message, receive the model's reply streamed token by token.
POST /v1/conversations -> { conversation_id }
POST /v1/conversations/{id}/messages
{ "content": "Explain CAP", "model": "large" }
-> 200 text/event-stream
data: {"type":"token","text":"CAP"}
data: {"type":"token","text":" says"} ...
data: {"type":"done","message_id":"m_9","usage":{"in":2048,"out":412}}
POST /v1/generations/{id}/cancel -> 204
GET /v1/conversations?cursor=... GET /v1/conversations/{id}/messages
High-Level Design (HLD)
The chat service rebuilds the prompt from history each turn and streams tokens back as the model produces them.
Chat service is the orchestrator: it authenticates, checks quota, loads recent history, builds the prompt (system prompt + trimmed history + new message), and streams the model's output back over SSE.
Inference gateway routes to a pool of model servers by model and load; model servers do continuous batching on GPUs (see LLM inference platform).
Conversation store keeps messages durably; the assistant message is saved as the stream completes (or partially, if cancelled).
Safety filters run on input and output; usage events feed billing and analytics asynchronously.
Low-Level Design (LLD)
parent_id lets a regenerated answer branch from the same user message; usage feeds quotas and billing.
Building the prompt: the context window
budget = model_context (e.g. 128K tokens) - reserved_for_answer (e.g. 4K)
prompt = [system prompt]
+ [summary of older turns] # when history exceeds the budget
+ [most recent turns, newest last] # as many as fit
+ [new user message]
Summaries are produced asynchronously and stored on the conversation, so
long chats don't resend (or re-pay for) their whole history every turn.
Providers cache the key/value attention state for a repeated prefix ("prompt caching"), so keeping the system prompt and older history byte-identical between turns makes each new turn cheaper and faster.
Streaming end to end
Model server emits tokens as they're generated; the chat service forwards them on an SSE connection, buffering in memory, and persists the full message at the end.
Cancel: the client calls cancel (or closes the stream); the chat service tells the model server to stop that sequence, freeing its batch slot, and saves the partial answer.
Reconnects: each generation has an id; a client that reconnects can fetch the finished message from the store instead of regenerating.
Data model
conversations(conversation_id PK, user_id, title, model, summary, updated_at)
-- index (user_id, updated_at DESC) for the sidebar list
messages(conversation_id, message_id, parent_id, role, content, tokens, created_at,
PRIMARY KEY (conversation_id, message_id))
-- partitioned by conversation_id: one conversation = one partition, read in order
usage(user_id, day, tokens_in, tokens_out) -- quotas and billing
Capacity and overload
When GPUs are saturated, the gateway queues requests with a short deadline, sheds free-tier traffic first, or falls back to a smaller model; per-user token-bucket limits stop one user from monopolising capacity.
Follow-up Questions
Why SSE and not WebSockets?
Each answer is a one-way stream in response to one request. SSE is plain HTTP, works through proxies and load balancers, reconnects automatically, and needs no bidirectional protocol.
How do you keep costs down?
Continuous batching on GPUs, prompt/prefix caching, trimming and summarising history, routing simple requests to smaller models, and caching answers to identical popular prompts where appropriate.
A GPU node dies mid-answer. What does the user see?
The stream errors; the client offers "regenerate", which re-sends the same prompt to a healthy replica. Generation state isn't worth replicating: it's cheaper to redo.
How would you add retrieval over the user's documents?
Before calling the model, retrieve relevant chunks from a vector index scoped to the user and add them to the prompt: that is RAG, covered in the RAG chatbot design.