Classic Case Studies

Design a Web Crawler

Watch: "Design a Web Crawler System Design Interview w/ a Ex-Meta Staff Engineer" — Hello Interview

In simple words

A tireless reader in a huge library who reads a page, writes down every reference it mentions on a to-read list, and repeats, while never visiting the same shelf twice and never pestering one librarian too often.

Problem Statement

Design a web crawler that downloads billions of web pages for a search engine: start from seed URLs, fetch pages, extract links, and keep discovering and re-fetching pages, while being polite to websites, avoiding duplicates and traps, and scaling across many machines.

Functional Requirements

Given seed URLs, crawl pages and discover new URLs from their links.
Store page content (HTML) for indexing.
Respect robots.txt and per-site politeness (rate limits).
Re-crawl pages periodically, more often for pages that change often.
Out of scope: rendering JavaScript-heavy pages (mention), indexing and ranking.

Non-Functional Requirements

Scale: 1B pages/month.
Politeness: never overload a site (e.g. at most one request per second per host).
Robustness: survive malformed HTML, slow servers, crawler traps, machine failures.
Extensibility: new content types (PDFs, images) can be added later.

1B pages / month ~ 400 pages/s (peak ~800/s)
avg page 100 KB -> 100 TB/month raw HTML (compress ~5x -> ~20 TB)
URLs seen: tens of billions over time -> dedup set must be compact (Bloom filter / hashed ids)

Core Entities

URL (normalised) with metadata: last crawled, next due, priority, host.
Host: robots rules, crawl delay, next allowed fetch time.
Page: url, fetched_at, status, content hash, content location in blob storage.

API Design

Mostly internal components rather than a public API:

Frontier.add(urls[], priority)          Frontier.next(worker_id) -> url (host-polite)
Fetcher.fetch(url) -> {status, headers, body}
Store.put(url, content, hash)           Seen.check_and_add(url) -> bool

High-Level Design (HLD)

Fetch, store, extract links, drop the ones already seen, and feed the rest back into the frontier.

seeds -> [ URL Frontier ] --(url)--> Fetcher workers --(DNS cache, robots cache)
           ^  priority queues            |
           |  + per-host queues          v  HTML
           |                      [ content store: object storage ]
           |                             |
           |                      Parser / link extractor
           |                             |  links
           |                      URL normaliser -> URL filter (robots, type, traps)
           |                             |
           +------ new URLs ------ "seen?" check (Bloom filter + URL db)
                                  content dedup: hash / simhash of page

Low-Level Design (LLD)

Important URLs first, but each host has its own queue and a minimum delay between fetches.

The URL frontier: priority + politeness

Front queues (priority):  F1 (high) ... Fn (low)
   priority from PageRank-like importance, change frequency, freshness need
                 |  prioritiser picks from front queues (weighted)
                 v
Back queues (politeness): one FIFO per host (or host group)
   host -> back queue mapping table
   a heap of (next_allowed_time, back_queue) decides which host is due

worker: pop the earliest-due host from the heap -> take its next URL ->
        fetch -> put host back with next_allowed_time = now + crawl_delay

This two-level design gives important pages first while guaranteeing no host gets more than one in-flight request at a time. The frontier is partitioned by host hash across machines so each host's queue lives in one place.

Dedup and normalisation

Normalise URLs: lowercase host, remove default ports and fragments, sort/strip tracking query parameters.
URL seen-check: a Bloom filter in memory (few false positives mean a few pages skipped, acceptable) backed by a persistent store of URL hashes.
Content dedup: exact duplicates by content hash; near-duplicates (mirrors, boilerplate changes) by SimHash/MinHash fingerprints.

Robustness

Timeouts and maximum page size on fetches; retry transient errors with backoff; record permanent failures (404/410) and slow down on 429/503.
Crawler traps (infinite calendars, session ids in URLs): cap URL length and path depth, limit pages per host per crawl cycle.
Cache DNS results and robots.txt per host (DNS lookups are otherwise a big latency cost).
Frontier state is checkpointed to disk so a crashed worker's URLs are re-queued.

Re-crawl scheduling

Track how often each page's content hash changes and set its next due time accordingly: news homepages every few minutes, static docs every few weeks. Honour Last-Modified/ETag with conditional GETs to save bandwidth.

Follow-up Questions

How do you distribute crawling across 100 machines?: Partition by hash(host): each machine owns a set of hosts, their queues and politeness state. Discovered URLs for other hosts are forwarded to the owning machine. Consistent hashing eases adding machines.
How do you crawl JavaScript-rendered pages?: A separate, much more expensive rendering tier (headless browsers) for pages detected as needing it, with its own lower throughput budget.
Why a Bloom filter for seen URLs instead of a hash set?: With tens of billions of URLs, an exact in-memory set needs hundreds of GB; a Bloom filter at ~1% false positives needs roughly 10 bits per URL, and a false positive only means skipping a page.