Classic Case Studies

Design a Notification System

In simple words

A post office for your app: other services drop off letters ("order shipped"), the post office checks each person's preferences (SMS, email, app), queues the letters by urgency, and hands them to the right courier, retrying when a courier fails.

Problem Statement

Design a system that sends notifications to users over push (iOS/Android), SMS, email and in-app channels. Other services (orders, payments, social, marketing) ask it to notify users; it decides the channel, respects user preferences, and delivers reliably at scale without spamming anyone.

Functional Requirements

Send a notification to one user or a large segment, on one or more channels.
Templates with variables (e.g. "Your order {{id}} has shipped").
User preferences and opt-outs per channel and category; quiet hours.
Scheduled sends; delivery status tracking.
Out of scope: building the actual push/SMS/email networks (we use providers: APNs, FCM, Twilio, SES).

Non-Functional Requirements

Reliability: transactional notifications (OTP, payment) must not be lost; at-least-once with dedup.
Latency: OTPs within seconds; marketing can take minutes.
Scale: e.g. 50M notifications/day, bursts of 10M for a campaign.
Isolation: a marketing blast must not delay OTPs; one provider outage must not block others.

50M/day ~ 600/s average; campaigns burst to ~50K/s for minutes
-> queue-based, horizontally scaled workers; providers have their own rate limits

Core Entities

Notification: id, user_id, category, template_id, params, channels, priority, status, idempotency_key.
Template: per channel and locale.
UserPreference: per user, category and channel: enabled, quiet hours.
DeviceToken / Contact: push tokens per device, email, phone.
DeliveryAttempt: notification_id, channel, provider, status, error, timestamps.

API Design

POST /v1/notifications
  { "user_id": 42, "category": "order_update", "template": "order_shipped",
    "params": { "order_id": "A123" }, "channels": ["push","email"],
    "priority": "high", "send_at": null }
  Idempotency-Key: order-A123-shipped
  -> 202 { "notification_id": "n_789" }

POST /v1/campaigns { segment_id, template, channels, send_at }  -> 202
GET  /v1/notifications/{id}  -> { status per channel }
PUT  /v1/users/{id}/preferences { ... }

High-Level Design (HLD)

Separate queues per priority and channel keep a marketing blast from delaying an OTP.

services --> Notification API --> validate, dedupe (idempotency key), persist
                    |
                    v
            [ priority queues ]  high (OTP/transactional)   low (marketing)
                    |
            Router / Preference workers
              - load user prefs, quiet hours, contact info
              - render template per channel & locale
              - per-channel rate limits (don't spam a user)
                    |
      +-------------+-------------+-------------+
      v             v             v             v
 [push queue]  [sms queue]  [email queue]  [in-app queue]
      |             |             |             |
 push workers   sms workers  email workers  in-app store + WebSocket
      |             |             |
    APNs/FCM     Twilio/...   SES/SendGrid   (with provider failover)
      \_____________ delivery status / receipts ____________/
                          -> status DB + analytics

Separate queues by priority and by channel so one slow channel or one campaign can't block others.
Campaigns are expanded into per-user jobs gradually (segment pagination), not all at once.

Low-Level Design (LLD)

The idempotency key's unique constraint stops duplicates; attempts record every provider call for retries and status.

Data model

notifications(id PK, user_id, category, template_id, params JSON, priority,
              status, idempotency_key UNIQUE, created_at, send_at)
delivery_attempts(id PK, notification_id, channel, provider, status, error, attempt_no, ts)
preferences(user_id, category, channel, enabled, PRIMARY KEY(user_id, category, channel))
devices(user_id, device_id, platform, push_token, last_seen, PRIMARY KEY(user_id, device_id))

Worker flow (per channel)

msg = queue.receive()
if already_sent(msg.notification_id, channel): ack; return   # dedupe
try:
    provider.send(render(msg))               # with timeout
    record(status="SENT"); ack
except TransientError:
    requeue with exponential backoff (attempt_no + 1)
except PermanentError (bad token / unsubscribed):
    record(status="FAILED"); deactivate token; ack
after N attempts -> dead-letter queue + alert

Provider failover: a circuit breaker per provider; route SMS to a secondary provider when the primary's error rate spikes.
Frequency capping: per-user counters (e.g. at most 3 marketing pushes/day) in Redis.
Scheduled sends: store with send_at; a scheduler polls due rows (or uses delayed queues) and enqueues them.
Invalid tokens: APNs/FCM report uninstalled apps; delete those tokens to keep pushes cheap.

Follow-up Questions

How do you guarantee an OTP isn't sent twice when the caller retries?: The caller passes an idempotency key; the API stores it with a unique constraint and returns the existing notification on retries. Workers also dedupe by (notification_id, channel) before calling the provider.
A 20M-user campaign is running. How do you keep OTPs fast?: Separate high-priority queues and dedicated workers for transactional traffic, rate-limit campaign expansion, and give providers separate quotas/accounts per traffic class.
Push provider is down for 10 minutes. What happens?: Messages stay in the push queue and retry with backoff; the breaker stops hammering the provider; for critical categories, fall back to SMS after a timeout if the user allows it.