A message queue is a buffer between services: producers put messages (tasks or events) in, and consumers take them out and process them later, at their own pace. The producer doesn't wait for the work to be done. Examples: RabbitMQ, Amazon SQS, Apache Kafka (a log rather than a classic queue), Google Pub/Sub.
In simple words
A restaurant's order rail. Waiters clip order tickets on the rail and go back to customers; cooks take tickets off the rail at their own pace. A rush of orders doesn't crash the kitchen, it just makes the rail longer for a while.
The producer answers immediately; workers process at their own pace and failed messages are set aside.
Why It Exists
Decoupling: the producer doesn't need to know who consumes or whether they are up right now.
Asynchrony: slow work (sending email, encoding video, generating thumbnails) leaves the request path, so users get a fast response.
Load levelling: a spike of 50K events/s can be absorbed and processed at 5K/s instead of crashing the consumer.
Reliability: messages persist until processed; a crashed consumer's work is redelivered.
How It Works
API server --publish--> [ queue: m5 m4 m3 m2 m1 ] --> worker 1
(returns 202 --> worker 2
immediately) --> worker 3
worker: receive m1 -> process -> ACK -> queue deletes m1
crash before ACK -> after a visibility timeout m1 is redelivered
fails N times -> m1 moves to a dead-letter queue (DLQ) for inspection
Types
Point-to-point (work queue): each message goes to one consumer. Used to spread tasks across workers.
Publish/subscribe: each message goes to every subscribed group (one event, many reactions: email, analytics, search indexing).
Log-based (Kafka): messages are kept in an append-only, partitioned log for a retention period; consumers track their own offset and can replay. Ordering is per partition.
Delivery Guarantees
At-most-once: may lose messages, never duplicates.
At-least-once: never loses, may duplicate (the usual default). Consumers must be idempotent.
Exactly-once: only within specific systems and boundaries; in practice achieved as at-least-once + idempotent processing.
Expect duplicates and out-of-order delivery; design consumers for it.
Monitor queue depth / consumer lag: a growing queue means consumers can't keep up.
Use a DLQ so one poison message doesn't block or loop forever.
Follow-up Questions
RabbitMQ/SQS vs Kafka?
Classic queues delete messages once acknowledged and suit task distribution with per-message acks and routing. Kafka keeps an ordered, replayable log per partition with very high throughput, suiting event streaming, multiple independent consumers and reprocessing history.
How do you keep order for one user's events?
Use a partition/message-group key such as user_id: all of that user's messages go to one partition and are consumed in order, while different users are processed in parallel.
Consumers fall behind and the queue keeps growing. What do you do?
Scale consumers (up to the partition count for Kafka), make processing faster or batched, shed or defer low-priority work, and apply backpressure upstream if needed.