Rate Limiting

Any public API needs a way to stop a single client — malicious or just buggy — from monopolizing shared capacity, e.g. capping a user at 100 requests/minute. This matters even more for AI APIs, where every request can trigger expensive LLM inference — an unthrottled client isn't just annoying, it's a direct cost problem. Rate limiting is a near-guaranteed system-design interview topic, and the choice of algorithm has real consequences for burstiness and fairness that interviewers expect you to articulate, not just name.

What Is Rate Limiting?

Rate limiting is a rule that caps how many times someone can do something in a given window of time — for example, "at most 100 requests per minute per user." Once a client hits that cap, the system starts rejecting or delaying its requests until the window resets, so one overly aggressive user can't hog resources that everyone else needs too.

Where This Is Used

• Twitter/X and Reddit's public APIs cap how many calls a single developer key can make per 15-minute window.
• Login pages rate-limit password attempts per account or IP, to slow down brute-force guessing.
• OpenAI, Anthropic, and other AI providers cap requests and tokens per minute per API key, partly because each request costs real compute.
• Ride-sharing apps like Uber throttle how often a driver's app can ping the server for location updates.

A Day-to-Day Example

Think of a popular bakery that only has one oven. To keep any one customer from buying out the whole day's bread, the bakery posts a sign: "Limit 4 loaves per customer per visit." Everyone still gets served, the oven never gets overwhelmed by a single bulk order, and the line keeps moving fairly for everyone behind them.

Deep Dive: Token Bucket, in Detail

A bucket has a fixed capacity (say, 100 tokens) and a refill rate (say, 10 tokens/sec). Every request consumes one token. If the bucket is empty, the request is rejected (or delayed). Crucially, this allows controlled bursts: a client that's been idle can fire up to 100 requests instantly (draining the full bucket), then is throttled back to the steady 10/sec refill rate — versus a naive "10 requests per second, hard cap" scheme that would reject a burst even from a well-behaved client that's simply been quiet.

Token Bucket — capacity=100, refill=10/sec

  [||||||||||||||||||||] 100 tokens available (bucket full, client idle)
  Client bursts 60 requests instantly:
  [||||||||||||........] 40 tokens left  -- all 60 succeeded, no throttling
  Client keeps firing at 50 req/sec:
  bucket drains toward 0, then requests get rejected once empty,
  refilling at 10/sec regardless of demand
  [....................] 0 tokens -- further requests get 429 until refill

Deep Dive: Comparing the Four Algorithms

• Fixed window — count requests in discrete windows (e.g. per calendar minute), reset to 0 at each boundary. Simplest to implement, but a client can send 100 requests in the last second of one window and another 100 in the first second of the next — 200 requests in ~1 second even though the limit is "100/minute." This boundary-burst flaw is a classic interview gotcha.
• Sliding window (log or weighted-counter approximation) — tracks requests over a continuously rolling window instead of resetting at fixed boundaries, closing the fixed-window loophole at the cost of more memory (storing timestamps, or interpolating between adjacent fixed windows).
• Token bucket — as above: allows bursts up to capacity, enforces an average rate over time. The most commonly cited answer for "design a rate limiter" because it cleanly expresses both a burst allowance and a sustained rate in two numbers.
• Leaky bucket — requests queue and are processed ("leak out") at a strictly fixed rate, regardless of how bursty the input is. This smooths output into a constant rate — good when the downstream truly cannot handle any burst at all — but it means a legitimate burst gets delayed/queued rather than let through, unlike token bucket.

Key Pointers

• Rate limiting caps how many requests a client (identified by IP, API key, or user ID) can make in a time window, protecting backends from abuse, buggy retry loops, controlling infrastructure cost, and ensuring fair resource usage across tenants.
• Token bucket allows bursts up to bucket capacity while enforcing an average rate via the refill rate — usually the default recommendation unless the downstream needs a hard constant-rate cap (leaky bucket).
• Fixed window is simple but allows up to ~2x the intended rate at a window boundary; sliding window avoids that at the cost of more state.
• In a distributed system with multiple API server instances, in-memory per-instance counters don't work — a client could get 100 requests through each of 5 instances for 500 total. A shared store like Redis (often using INCR + EXPIRE, or a Lua script for atomicity) holds the shared counter/token state so all instances enforce the same limit consistently.
• Enforcement point matters: at the API gateway/edge it's cheap and protects everything behind it but is coarse-grained; per-service or per-resource it's fine-grained but traffic has already consumed some capacity getting there.

How It Works

1. A request arrives tagged with a rate-limit key (IP, API key, user, or endpoint).
2. The limiter checks and updates state for that key — token count, window count, or queue position — often in a fast shared store like Redis for consistency across distributed instances.
3. If within the limit, the request proceeds and the limiter records the consumption.
4. If over the limit, the request is rejected (HTTP 429) or queued/delayed, typically with a Retry-After header telling the client when to try again.
5. State resets or refills over time according to the algorithm's rule — token refill, window rollover, or queue drain.

Try It Yourself

# Hammer an endpoint and watch 429s appear once you exceed its limit
for i in $(seq 1 150); do curl -s -o /dev/null -w "%{http_code} " https://example.com/api/resource; done; echo

# Inspect rate-limit headers most APIs expose
curl -sI https://api.example.com/v1/resource | grep -i "ratelimit\|retry-after"

# Load-test to find the actual enforced rate/burst behavior —
# compare a short sharp burst vs a sustained steady rate
hey -z 10s -c 200 -q 50 https://api.example.com/v1/resource   # burst-heavy
hey -z 10s -c 10 -q 5 https://api.example.com/v1/resource     # steady rate

# Or with ApacheBench for a quick burst test
ab -n 500 -c 100 https://api.example.com/v1/resource

# Observe Retry-After being honored — parse it and sleep accordingly
resp=$(curl -sI https://api.example.com/v1/resource)
echo "$resp" | grep -i retry-after

Interview Q&A

Q: Design a rate limiter for an API — what algorithm and why?

A: Token bucket, generally — it's the algorithm that most naturally expresses both a burst allowance and a sustained average rate with just two parameters (capacity and refill rate), matching how real client traffic behaves (bursty, not perfectly smooth). I'd key the bucket per API key or per user ID rather than per IP alone, since IPs can be shared (NAT, corporate networks) or spoofed/rotated.

Q: How do you make this work across multiple API server instances?

A: State has to be shared, not per-instance — a local in-memory counter on each server only limits traffic to that one instance, so a client hitting 5 instances behind a load balancer effectively gets 5x the intended limit. I'd put the counters/token state in Redis, using an atomic operation (a Lua script, or Redis's built-in atomic INCR+EXPIRE) so concurrent requests from different servers don't race and both succeed when only one should have.

Q: What's the downside of putting the rate limit check in Redis on every request?

A: It adds a network round-trip to Redis on the hot path of every request, which is extra latency and a new dependency that must itself be highly available — if Redis is down, you need a defined fallback behavior (fail open and allow traffic, or fail closed and reject it, depending on which failure mode is safer for that particular API). Some systems mitigate the round-trip cost with local caching of "definitely still has tokens" state and only hitting Redis near the limit boundary.

Q: Why might rate limiting matter more for an AI/LLM API than a typical CRUD API?

A: Because the cost per request is wildly asymmetric — a CRUD request might cost microseconds of DB time, while an LLM inference request can cost real compute (GPU time) and real dollars per call. An unthrottled or abused client isn't just a performance nuisance, it's a direct and potentially large infrastructure bill, which is why AI APIs commonly rate-limit by token count consumed, not just request count.

System Design Tie-in

"Design a rate limiter" is itself a standalone, frequently-asked system design question — the expected answer covers algorithm choice (token bucket, with sliding window as the fallback for stricter fairness), the distributed-state problem (Redis, atomicity), and where to enforce it (gateway vs per-service). It also shows up as a component inside bigger designs: any "design an API gateway" or "design a public-facing API platform" question should include rate limiting as a first-class concern, and any AI-product system design (chat API, image generation API) should specifically call out per-user or per-tier limits tied to compute cost, not just raw request count.

Trade-offs

Token bucket and leaky bucket smooth traffic more fairly than fixed windows but require more state to track; distributed rate limiting across many gateway instances needs a shared store like Redis, trading strict accuracy for either extra latency (synchronous checks) or approximate limits (local counters synced periodically). Interview gotcha: edge-level rate limiting protects infrastructure but can't see per-tenant business cost — production systems often combine a coarse gateway limiter with finer per-service or per-resource limits underneath.