Like a newspaper boy: when your friends post, their posts are dropped into your personal mailbox ahead of time, so opening the app is just reading your mailbox. For celebrities with millions of followers, you fetch their posts yourself instead of filling millions of mailboxes.
Design the home feed of a social network (Twitter/X, Instagram, Facebook): users post content and follow others, and each user's feed shows recent posts from the accounts they follow, newest or best first. The core difficulty is fan-out: one post must reach many feeds, and one feed read gathers posts from many authors.
Posts: 300M DAU x 0.5 posts/day = 150M/day -> ~1,700/s (peak ~5K/s)
Feed reads: 300M x 10 opens/day = 3B/day -> ~35K/s (peak ~100K/s)
Avg 300 followers -> push fan-out = 150M x 300 = 45B feed inserts/day (~500K/s)
-> reads dominate; precompute feeds, but celebrities need special handling
POST /v1/posts { text, media_ids[] } -> 201 { post_id }
POST /v1/follows { followee_id } -> 204
DELETE /v1/follows/{followee} -> 204
GET /v1/feed?cursor=<opaque>&limit=20 -> { items: [post...], next_cursor }
Cursor pagination (the last seen post id/time) rather than offsets: new posts arriving at the top would shift offsets and cause duplicates or gaps.
WRITE PATH
client -> Post service -> posts DB (sharded by post_id) + post cache
\-> publish PostCreated --> [ queue ]
|
Fan-out workers (per author)
- page through followers (graph service)
- skip if author is a "celebrity"
- LPUSH post_id into feed:{follower} (Redis),
trim to latest ~800
READ PATH
client -> Feed service
1. ids = feed:{user} from Redis (precomputed, pushed)
2. + recent posts of followed celebrities (pull, cached per celebrity)
3. merge, rank, take page
4. hydrate ids -> post objects (post cache, author cache)
5. media URLs point at the CDN
posts(post_id PK [snowflake, time-sortable], author_id, text, media JSON, created_at)
-- sharded by post_id; secondary index / table user_posts(author_id, post_id DESC)
follows(follower_id, followee_id, created_at)
-- two tables (or indexes): by follower (who do I follow) and by followee (my followers)
feed:{user_id} Redis list/sorted set of post_ids (score = time), capped ~800
celebrity:{author_id}:recent cached list of latest post_ids
def get_feed(user, cursor, limit=20):
pushed = redis.zrevrangebyscore(f"feed:{user}", max=cursor, count=limit*2)
pulled = []
for c in celebrities_followed(user): # usually a short list
pulled += recent_posts_cached(c, before=cursor)
ids = merge_by_time(pushed, pulled)[:limit]
posts = hydrate(ids) # multi-get from post cache
posts = [p for p in posts if visible_to(user, p)] # deleted, blocked, unfollowed
return posts, next_cursor(posts)
Chronological is simplest. A ranked feed fetches a larger candidate set (a few hundred ids), scores each with a model (affinity with author, engagement so far, recency, content type), and returns the top page; scores and candidates can be cached for a few minutes per user.