A message queue or event stream is a broker that sits between a producer and a consumer so the two do not have to run at the same speed, be up at the same time, or even know about each other. Interviewers probe it because "just put it on a queue" is the standard answer to almost every scaling question, and the follow-ups expose whether you understand what the broker actually guarantees: ordering (only within a partition), delivery (at-least-once in practice), and what happens when a consumer crashes, is slow, or meets a message it cannot process. For a Backend or AI engineer the same machinery carries embedding pipelines, batch inference jobs, feature updates and model-monitoring events, where a burst of documents or requests must be absorbed without melting a GPU pool or a rate-limited provider API. The wire-level view lives in Message Queues (Kafka, RabbitMQ, SQS); this page is about designing with them and about failure semantics. Pair it with Idempotency & Exactly-Once Semantics, because every retry story below ends there.
A message queue is a waiting line for work: one part of a system drops a task into the queue (like "resize this photo" or "send this email"), and a separate worker picks it up and does it whenever it's ready, without the two parts having to talk to each other directly or be running at the exact same moment. This lets a busy website accept a task instantly and process it a few seconds later in the background, instead of making the user wait for the whole job to finish.
Event streaming is a related idea where, instead of one task being handled by one worker and then thrown away, a continuous flow of "this happened" facts (a payment was made, a user signed up) gets recorded in order and kept around so that several different systems can read through that same history, even at different times or different speeds. Tools like Kafka are built for this: think of it less like a to-do list and more like a running news ticker that many different readers can tune into.
A message queue is like the order slips at a busy restaurant kitchen: the waiter clips a ticket to the line, and whichever cook is free takes the next ticket and cooks it, without the waiter and cook needing to coordinate in person. Event streaming is more like a post office's daily log of every package that came through a sorting facility: it's a running, ordered record that the delivery team, the billing department, and customer service can each look back through independently, at their own pace, for their own purposes.
QUEUE (SQS, RabbitMQ classic queues) LOG (Kafka, Redpanda, Kinesis)
producer --> [m5 m4 m3 m2 m1] --> worker A producer --> offset: 0 1 2 3 4 5
--> worker B [m0 m1 m2 m3 m4 m5] <- append-only
^ ^
a message is delivered to ONE worker, group "billing" ----+ | (offset 2)
then ACKed and DELETED. group "search" -------------+ (offset 4)
Broker tracks per-message state Each group keeps its own read position.
(in flight, visible, acked). Nothing is deleted on read; retention
(time/size) or compaction removes data.The structural difference is where the state lives. A queue broker tracks the state of every message (ready, in flight, acknowledged, dead-lettered), so it can hand out work to whichever worker is free, redeliver one specific message that timed out, and delay or prioritise individual messages. A log broker tracks almost nothing per message: it appends bytes to a partition and each consumer group stores a single number, its offset. That makes reads and writes sequential and cheap, which is why logs reach very high throughput, and it makes replay trivial (reset the offset), but it removes per-message operations: you cannot ack message 7 while leaving message 6 pending, and you cannot easily delay one message.
Rule of thumb: use a queue when the messages are tasks ("resize this image", "run this inference job") and you want a pool of workers with per-task retry; use a log when the messages are facts ("order 42 was paid", "document 9 was updated") that several independent systems will consume, possibly later, possibly again. Some products blur the line: RabbitMQ has streams, SQS FIFO adds ordering, Kafka can be made to behave like a work queue with enough partitions. Name the product and the version's feature set before claiming a behaviour, and check current docs.
topic "doc-events", 3 partitions, key = document_id
hash(key) % 3 = partition
doc-7 -> p1: [doc-7:created][doc-7:updated][doc-7:deleted] ordered
doc-9 -> p0: [doc-9:created][doc-9:updated] ordered
doc-3 -> p1: interleaved with doc-7 on the same partition
Ordering guarantee: total order INSIDE one partition. Across partitions: none.
Parallelism ceiling: within a consumer group, one partition is read by at most one consumer.
3 partitions -> at most 3 useful consumers; a 4th sits idle.
Changing 3 -> 6 partitions changes hash(key) % N: the same key can move to a new partition,
so per-key order can break across that boundary (in-flight old messages vs new ones).Partitions are the unit of both ordering and parallelism, and the two pull in opposite directions. Global ordering means one partition means one consumer, so throughput is capped by a single reader. The usual design gives up global order and keeps per-key order: pick the key so that events which must be applied in sequence (all events for one user, one document, one order) hash to the same partition. The failure modes are a hot key (one giant tenant overloads one partition while others idle), a poor partition count (too few caps consumer parallelism; too many increases broker metadata, file handles and rebalance time), and repartitioning later, which breaks the key-to-partition mapping. Size the partition count for the peak consumer parallelism you expect in a year, not for today.
Two ordering traps deserve a mention. First, a producer that retries a failed send can reorder messages unless the producer is configured to preserve order (Kafka's idempotent producer, or limiting in-flight requests); check your client's settings. Second, competing consumers on a plain queue give no cross-message ordering at all: worker B can finish message 2 before worker A finishes message 1. SQS FIFO and RabbitMQ (with a single consumer per queue) can preserve order, at the cost of throughput or head-of-line blocking.
Consumer group "embedder": members C1, C2 share partitions p0..p2
C1 owns p0, p1 C2 owns p2 (assignment done by the group coordinator)
committed offsets stored by the GROUP: p0=3 p1=0 p2=4 (Kafka: internal topic __consumer_offsets)
Poll -> process -> commit Commit -> process -> poll
crash between process and commit: crash between commit and process:
restart re-reads from old offset restart skips the message
=> processed TWICE (at-least-once) => processed ZERO times (at-most-once)
"Exactly-once" end to end = at-least-once delivery + idempotent effect (dedupe key, upsert,
transactional outbox). Kafka transactions give exactly-once only for read-process-write
WITHIN Kafka; an external side effect (email, DB, HTTP call) is not covered.The commit position is the whole delivery-semantics story. Commit after processing gives at-least-once: a crash between the side effect and the commit means the message is delivered again. Commit before processing gives at-most-once: a crash after the commit loses the message. Auto-commit on a timer is worse than either because the commit is decoupled from the processing, so you get whichever accident the timer produces. The practical default is at-least-once with idempotent handlers (see the demo below and Idempotency & Exactly-Once Semantics). On SQS the same idea appears as the visibility timeout: a received message is hidden for N seconds and reappears if you have not deleted it, so if the handler takes longer than the timeout you get a duplicate delivery while the first attempt is still running.
Group membership adds a second failure source. When a consumer joins, leaves or is deemed dead (missed heartbeats, or exceeded Kafka's max.poll.interval.ms because processing was too slow), the group rebalances and partitions move to other members; during a rebalance consumption pauses for the affected partitions (newer cooperative protocols reduce this), and any uncommitted work on the old owner will be redone by the new one. Long-running inference calls inside a poll loop are a classic way to trigger rebalance storms: keep the poll loop fast, hand work to a bounded worker pool, and commit only offsets whose work is complete.
The log connects naturally to databases: change data capture turns a database's own write-ahead log into such a topic, which is how a search index or vector store is kept in sync with the source of truth without dual writes. See Change Data Capture (CDC).
+--------+ fail +-----------+ fail +-----------+ N failures +-------+
topic ---> | handle | ------> | retry-1m | ------> | retry-10m | -----------> | DLQ |
+--------+ +-----------+ +-----------+ +-------+
| |
success alert, inspect, fix,
v replay (redrive)
commit
Poison message: fails EVERY time (bad schema, oversized payload, a bug on one input).
Without a DLQ, a log consumer retrying forever blocks its whole partition (head-of-line blocking).
Rule: retry with backoff + jitter for TRANSIENT errors; dead-letter after a bounded number of tries
for PERMANENT ones; never let one message hold the partition.Retries must distinguish transient failures (timeout, 429, broker blip) from permanent ones (validation error, unparsable payload). Transient errors get exponential backoff with jitter, otherwise every consumer retries in lock-step and hammers the recovering dependency. Permanent errors should go straight to a dead-letter queue with the error and original metadata attached. SQS and RabbitMQ support this natively (SQS via a redrive policy with a maximum receive count; RabbitMQ via dead-letter exchanges); Kafka has no per-message ack, so the application or framework publishes to a retry topic and a dead-letter topic itself. A DLQ is only useful if something watches it: alert on its depth and keep the payload replayable, because a silent DLQ is just data loss with better branding.
Backpressure is the question "what happens when the consumer is slower than the producer?". A queue absorbs the difference, but only until retention or disk limits are hit, and until end-to-end latency becomes unacceptable: a message that waits 40 minutes in the queue may be useless by the time it is processed. Consumer lag (log end offset minus committed offset, or approximate queue depth and age of the oldest message) is the metric that shows it. Watch the rate of change and the age, not only the absolute number. The toy simulation below uses ILLUSTRATIVE rates: a 30-second burst at 500 msg/s over a 100 msg/s baseline, and workers that each handle 60 msg/s.
# Lag under a traffic burst. ILLUSTRATIVE rates, 1 tick = 1 second.
produce = lambda t: 500 if 10 <= t < 40 else 100 # burst of 500 msg/s for 30 s, else 100 msg/s
def simulate(workers, per_worker=60, ticks=120):
lag, peak, drained_at = 0, 0, None
for t in range(ticks):
lag += produce(t)
lag -= min(lag, workers * per_worker) # each worker handles 60 msg/s
peak = max(peak, lag)
if t >= 40 and lag == 0 and drained_at is None:
drained_at = t
return peak, drained_at, lag
for w in (2, 4, 6, 10):
peak, drained, final = simulate(w)
cap = w * 60
print(f"workers={w:2d} capacity={cap:3d}/s peak_lag={peak:5d} drained_at={drained} final_lag={final}")
# Output:
# workers= 2 capacity=120/s peak_lag=11400 drained_at=None final_lag=9800
# workers= 4 capacity=240/s peak_lag= 7800 drained_at=95 final_lag=0
# workers= 6 capacity=360/s peak_lag= 4200 drained_at=56 final_lag=0
# workers=10 capacity=600/s peak_lag= 0 drained_at=40 final_lag=0With 2 workers capacity (120/s) barely exceeds baseline traffic, so after the burst the backlog drains at only 20 msg/s and is still huge at the end of the run. With 4 workers it does drain, but late; 10 workers (600/s, above the burst rate) never build a backlog, but that is capacity paid for 24 hours a day. The queue converts a peak-rate problem into an average-rate problem plus a latency budget, and the design question is how much delay the business tolerates. Options when lag grows: autoscale consumers on lag (bounded by the partition count for a log), shed or downgrade low-priority work, apply producer-side rate limits (see Rate Limiting & Throttling), and use bounded queues so overload is explicit instead of silent.
upload --> [docs topic] --> chunker --> [chunks topic, key=doc_id] --> embed workers (GPU / provider API)
|
[embeddings topic] <-- batch of 32-128 chunks per call
|
vector DB upsert (id = hash(doc_id + chunk_index + text))
|
status/lag metrics --> autoscaler
Design points
- work item = small and idempotent (a chunk id), not a giant blob; store payloads in object storage
- consumer batches messages to fill the GPU (see batching), but commits only after the upsert
- concurrency limit per provider (rate limit / quota) enforced in the consumer, not by hoping
- long jobs: extend the visibility timeout / heartbeat, or the broker will redeliver a running job
- separate queues (or priority topics) for interactive and bulk traffic so a re-index cannot starve usersQueues fit AI workloads because the expensive resource (GPU seconds, or a metered embedding API) is scarce and slow, so you want it to be always busy but never overloaded. A queue lets producers, such as a crawler that bulk-loads a million documents, run at their own speed while a bounded pool of workers pulls at the rate the GPU or quota permits. Two AI-specific pitfalls: message size and duration. Brokers are tuned for small messages (SQS has a hard message size limit, so check current docs; Kafka defaults are around 1 MB), so pass references to objects rather than documents, and remember that a 60-second inference call may outlive a default visibility timeout or poll interval. For request-time interactive inference a queue usually adds unwanted latency; it belongs to the asynchronous path, see Synchronous vs Asynchronous Processing.
A toy partitioned log with keys, a consumer group's committed offsets, a crash between processing and commit, a poison message and a dead-letter list, followed by the same run with an idempotent handler. It is a teaching model: no replication, no rebalance protocol, in-memory only. The stable zlib.crc32 hash is used so the partition assignment is reproducible.
import zlib
from collections import defaultdict
class Topic:
"""Partitioned append-only log. Messages are never deleted on read."""
def __init__(self, partitions):
self.parts = [[] for _ in range(partitions)]
def send(self, key, value):
p = zlib.crc32(key.encode()) % len(self.parts) # same key -> same partition
self.parts[p].append((key, value))
return p, len(self.parts[p]) - 1
class Group:
"""Consumer group: committed offset per partition, owned by the group (not the consumer)."""
def __init__(self, topic):
self.topic = topic
self.committed = defaultdict(int) # partition -> next offset to read
def lag(self):
return {p: len(log) - self.committed[p] for p, log in enumerate(self.topic.parts)}
def consume(group, partitions, handler, crash_after=None, max_retries=2, dlq=None):
"""Poll -> process -> commit AFTER processing (at-least-once)."""
handled = 0
for p in partitions:
log = group.topic.parts[p]
while group.committed[p] < len(log):
off = group.committed[p]
key, value = log[off]
for attempt in range(1, max_retries + 2):
try:
handler(p, off, key, value)
break
except ValueError as e:
if attempt > max_retries: # poison message: park it, keep moving
dlq.append((p, off, key, value, str(e)))
handled += 1
if crash_after is not None and handled == crash_after:
print(f" !! consumer crashed after handling p{p}@{off}, BEFORE committing it")
return False
group.committed[p] = off + 1 # commit offset
return True
t = Topic(3)
for i in range(3):
for user in ("user-a", "user-b", "user-c", "user-d"):
t.send(user, f"{user}-event-{i}")
t.send("user-b", "POISON")
print("partition sizes :", [len(x) for x in t.parts])
for p, log in enumerate(t.parts):
print(f" p{p}:", [v for _, v in log])
g = Group(t)
print("lag at start :", g.lag())
side_effects, dlq = [], []
def handler(p, off, key, value):
if value == "POISON":
raise ValueError("cannot parse payload")
side_effects.append((p, off, value)) # e.g. "send email", "write embedding"
print("\n-- run 1: C1 owns p0,p1 and crashes mid-way; C2 owns p2 and runs fine --")
consume(g, [0, 1], handler, crash_after=4, dlq=dlq)
consume(g, [2], handler, dlq=dlq)
print(" committed :", dict(g.committed), "| lag:", g.lag())
print("\n-- run 2: C1 restarts, resumes from the group's committed offsets --")
consume(g, [0, 1], handler, dlq=dlq)
print(" committed :", dict(g.committed), "| lag:", g.lag())
counts = defaultdict(int)
for _, _, v in side_effects:
counts[v] += 1
dupes = {v: n for v, n in counts.items() if n > 1}
print("\nside effects :", len(side_effects), "for", len(counts), "distinct events")
print("redelivered :", dupes)
print("DLQ :", dlq)
print("\n-- same run, idempotent handler (dedupe on partition+offset) --")
g2, seen, effects2, dlq2 = Group(t), set(), [], []
def idem(p, off, key, value):
if value == "POISON": raise ValueError("cannot parse payload")
if (p, off) in seen: return # already applied
seen.add((p, off)); effects2.append(value)
consume(g2, [0, 1], idem, crash_after=4, dlq=dlq2)
consume(g2, [2], idem, dlq=dlq2)
consume(g2, [0, 1], idem, dlq=dlq2)
print("side effects :", len(effects2), "distinct:", len(set(effects2)))
# Output:
# partition sizes : [3, 6, 4]
# p0: ['user-a-event-0', 'user-a-event-1', 'user-a-event-2']
# p1: ['user-c-event-0', 'user-d-event-0', 'user-c-event-1', 'user-d-event-1', 'user-c-event-2', 'user-d-event-2']
# p2: ['user-b-event-0', 'user-b-event-1', 'user-b-event-2', 'POISON']
# lag at start : {0: 3, 1: 6, 2: 4}
#
# -- run 1: C1 owns p0,p1 and crashes mid-way; C2 owns p2 and runs fine --
# !! consumer crashed after handling p1@0, BEFORE committing it
# committed : {0: 3, 1: 0, 2: 4} | lag: {0: 0, 1: 6, 2: 0}
#
# -- run 2: C1 restarts, resumes from the group's committed offsets --
# committed : {0: 3, 1: 6, 2: 4} | lag: {0: 0, 1: 0, 2: 0}
#
# side effects : 13 for 12 distinct events
# redelivered : {'user-c-event-0': 2}
# DLQ : [(2, 3, 'user-b', 'POISON', 'cannot parse payload')]
#
# -- same run, idempotent handler (dedupe on partition+offset) --
# !! consumer crashed after handling p1@0, BEFORE committing it
# side effects : 12 distinct: 12Read the output: after C1 crashed, the group's committed offset for p1 stayed at 0 even though C1 had already applied user-c-event-0, so the restarted consumer applied it again, giving 13 side effects for 12 distinct events. The poison message was tried three times, then parked in the DLQ with its partition and offset, while everything behind it kept flowing. Note that user-c and user-d share partition p1, so their events are interleaved but each user's own order is preserved. The idempotent variant fixes the duplicate by remembering (partition, offset); in real systems that memory must be durable and atomic with the side effect (a unique key in the same database transaction, or an upsert), otherwise the dedupe set is lost in the same crash. Now the equivalent real configuration, labelled as config, not output:
# Kafka consumer (kafka-python style / Java client property names), config sketch - not executed here
group.id = embedder
enable.auto.commit = false # commit manually after the upsert succeeds
auto.offset.reset = earliest # what to do when the group has no committed offset
max.poll.records = 64 # bound the work per poll
max.poll.interval.ms = 300000 # must exceed worst-case time to process one batch
isolation.level = read_committed # only relevant if producers use Kafka transactions
# SQS (boto3-style) sketch
VisibilityTimeout = 120 # > worst-case processing time; extend for long jobs
RedrivePolicy = {"deadLetterTargetArn": "<dlq-arn>", "maxReceiveCount": 5}Q: When would you choose a queue like SQS or RabbitMQ over a log like Kafka?
A: When messages are independent tasks handled by a pool of workers, you want per-message retry, delay or priority, and you do not need replay or several independent consumers of the same data. When several systems must consume the same events, when you need replay, or when throughput is very high and per-key ordering matters, a log fits better. The cost of the log is operational complexity (partitions, offsets, retention, rebalancing) and the loss of per-message ack and delay.
Q: What ordering does Kafka guarantee?
A: Order within a single partition only. Messages with the same key land on the same partition, so you get per-key ordering, but there is no order across partitions and none across topics. If you need strict order for all events you are limited to one partition and one consumer per group, which caps throughput.
Q: How do you get "exactly-once" processing?
A: You cannot get exactly-once delivery across a network in general; you get at-least-once delivery and make the effect idempotent, so a duplicate does nothing. Concretely: a unique message or business id with a dedupe table or upsert applied in the same transaction as the side effect. Kafka's transactions and idempotent producer give exactly-once for read-process-write inside Kafka, but an email or an external API call still needs its own idempotency key.
Q: A consumer keeps failing on one message. What happens, and what do you do?
A: On a log with in-order commits the partition is stuck behind it (head-of-line blocking), and lag grows on that partition only. Retry a bounded number of times with backoff for transient causes, then publish the message to a dead-letter topic with error context, commit past it, and alert on DLQ depth. The trade-off is that skipping a message can break per-key order for that key, so if strict order matters you may choose to halt and page instead of skipping.
Q: Consumer lag is climbing. How do you diagnose and fix it?
A: First separate rate from stall: is the producer rate higher than consumer throughput, or is one partition or one consumer stuck (poison message, rebalance loop, slow downstream)? Per-partition lag shows a hot key or stuck partition; uniform lag shows insufficient capacity. Fixes are more consumers (up to the partition count), batching, fixing the slow dependency, or shedding low-priority work, and beyond the partition limit you must repartition, which has ordering consequences.
Q: Why can adding more consumers not speed up a Kafka consumer group?
A: Each partition is consumed by at most one member of a group, so with 6 partitions the seventh consumer is idle. You need more partitions (with the key remapping caveat), or you must parallelise inside a consumer with a worker pool, at the cost of more complex offset tracking because commits must only advance past finished messages.
Q: How would you design an embedding pipeline for a bulk re-index of 20 million chunks?
A: Put chunk ids (not text) on a topic keyed by document, run a bounded pool of embedding workers that batch chunks for the GPU or provider API, and upsert vectors by a deterministic id so replays are harmless. Commit only after the upsert, cap concurrency by the provider's quota, send failures to a DLQ, and put bulk jobs on a separate queue so live ingestion and interactive traffic are not starved.
Follow-up: The handler takes longer than the visibility timeout on SQS. What goes wrong?
A: The message becomes visible again while the first worker is still running, so a second worker starts the same job and you have concurrent duplicates, possibly repeated after each timeout. Set the timeout above the worst-case duration, extend it periodically for long jobs, and keep the handler idempotent because a timeout can always still race with a slow success.
Follow-up: You need to add partitions to a live topic keyed by user id. What are the risks?
A: The hash-to-partition mapping changes, so a user's new events may go to a different partition than their unconsumed old ones, and two consumers can process them concurrently and out of order. Mitigate by draining the old partitions first, creating a new topic with the desired count and migrating consumers, or choosing a partition count with headroom in advance.
Follow-up: Kafka "exactly-once semantics" is enabled. Can a downstream Postgres write still be duplicated?
A: Yes. Kafka's guarantee covers the atomic combination of consuming offsets and producing to Kafka topics. A write to an external database is outside that transaction, so a crash between the SQL commit and the offset commit repeats the write. The fixes are an idempotent write (upsert or a unique key) or storing the offset in the same database transaction as the result.
In a design interview, reach for a broker when there is a rate mismatch, a fan-out, or a need to decouple failure domains: image or document processing, notification delivery, feature and embedding updates, audit and analytics events, and change streams from a database (see CDC). State assumptions and do the arithmetic. ILLUSTRATIVE: 1M documents times 20 chunks is 20M chunks; if one embedding worker sustains 200 chunks/s, 10 workers give 2,000/s and the bulk job takes 10,000 s, about 2.8 hours, and you would want at least 10 to 20 partitions so consumers can scale. For event traffic, assume 5,000 events/s at 1 KB average, roughly 5 MB/s or about 430 GB per day before replication; with 7 days retention and replication factor 3 that is on the order of 9 TB, which is a storage line item, not a rounding error. Also name the guarantees you are relying on (at-least-once, per-key order), the DLQ and lag alerting, and where the transactional boundary sits between the database and the broker (the outbox pattern, so a row and its event are committed atomically). If the broker itself needs a coordinator or metadata quorum, that is the consensus layer discussed in Consensus & Coordination.
Introducing a broker buys decoupling, burst absorption, independent scaling and replay, and it costs extra latency, another stateful system to run and monitor, and a set of new failure modes: duplicate deliveries, reordering across partitions, poison messages, rebalance storms, and silent lag. Eventual consistency becomes visible to users (the document is uploaded but not yet searchable), so the product must tolerate that. A queue is the wrong choice when the caller needs the answer immediately (interactive inference, a price quote), when the volume is small enough that a database table with SELECT ... FOR UPDATE SKIP LOCKED is simpler and transactional with your data, or when the team cannot yet operate Kafka. A log is the wrong choice for plain task distribution with per-message retries and delays, where a queue is simpler, and picking a log for a low-volume system usually buys operational burden without benefit. Whatever you choose, budget for idempotency, DLQ handling and lag monitoring from day one, because retrofitting them after the first duplicate charge or lost message is far more expensive.