Foundations

Scalability — Vertical vs Horizontal, Stateless Services & Bottlenecks

Scalability is a system's ability to handle more load by adding resources, ideally with performance and cost growing in proportion. It is not the same as speed: a fast system can fail to scale (it collapses past one machine), and a scalable system can be slow per request. Almost every system design interview is a conversation about which part of a design stops scaling first and what you do about it.

In simple words

A busy restaurant can buy a bigger stove (scale up) or hire more cooks with more stoves (scale out). More cooks works best when any cook can make any dish because recipes are on the wall, not in one cook's head. That is what "stateless" means.

State lives in shared stores, so any app server can handle any request and you scale by adding servers.

Deep Dive: Vertical vs Horizontal Scaling

Vertical (scale up)                 Horizontal (scale out)
  one bigger machine                  more machines of the same size
  +---------+      +-------------+    +---+      +---+ +---+ +---+ +---+
  |  4 CPU  |  ->  |   64 CPU    |    | A |  ->  | A | | B | | C | | D |
  |  16 GB  |      |   512 GB    |    +---+      +---+ +---+ +---+ +---+
  +---------+      +-------------+                 ^ load balancer ^

  + no code changes, no distribution    + near-unlimited ceiling
  + strong consistency stays easy       + failure of one node is survivable
  - hard ceiling (biggest box)          - needs stateless services or
  - single point of failure               partitioned data
  - price grows faster than capacity    - distributed-systems problems

In practice you do both: scale up until it's uneconomic or risky, then scale out. Databases are the classic case: a single primary scaled vertically goes a long way (and keeps transactions simple), then read replicas take read load, and only then do you shard writes.

Deep Dive: Stateless Services Are What Make Scale-Out Easy

A service is stateless when any instance can handle any request because nothing about the user lives in that instance's memory between requests. Session data, carts and uploads-in-progress go to a shared store (Redis, a database, object storage) instead. Then the load balancer can send each request anywhere, you can add or remove instances freely, and a crashed instance loses nothing.

Stateful (hard to scale)                 Stateless (easy to scale)
  user -> server A (session in RAM)        user -> any server -> Redis (session)
  A dies -> user logged out                any server dies -> retry elsewhere
  needs sticky sessions                    plain round-robin works

State does not disappear; it moves to the data tier, which is why "how do we scale the database?" is always the next question.

Deep Dive: Finding the Bottleneck

A system scales only as far as its tightest component. The usual suspects, in roughly the order they bite:

The database primary: all writes funnel into one node. Remedies: caching and read replicas for reads; batching, queues and finally sharding for writes.
Hot keys: one celebrity account, one viral link or one popular product gets a disproportionate share of traffic, so adding nodes does not help that key. Remedies: replicate the hot item in cache, split the key, or treat it specially.
Synchronous chains: A calls B calls C; latencies add and one slow dependency stalls everything. Remedies: timeouts, async processing via queues, parallel calls, caching.
Fan-out: one action causes N units of work (a post delivered to 10M followers). Remedies: see push vs pull.
Coordination: locks, distributed transactions and global counters serialize work. Remedies: partition the state so each piece has one owner.

Key Pointers

Scale up first when it's cheap; scale out when you hit the ceiling or need fault tolerance.
Make app servers stateless; push state into stores designed to be shared and replicated.
Reads scale with caches and replicas; writes scale with partitioning (sharding) and asynchrony.
Measure: throughput, p99 latency and saturation (CPU, connections, queue depth) show where the bottleneck is.
Horizontal scale buys fault tolerance too: many small nodes survive one failure; one big node does not.

How It Works

Put a load balancer in front of identical, stateless app servers; add servers as load grows (autoscaling on CPU or request rate).
Add a cache for hot reads and a CDN for static and media content.
Add read replicas when read load on the database grows.
Move slow or bursty work behind a message queue so the request path stays fast.
When a single primary can't take the write load or data size, shard by a key that spreads load evenly.

Follow-up Questions

Why not just buy a bigger database server?: Often you should, up to a point: it keeps transactions and queries simple. But there is a hard ceiling, cost grows faster than capacity at the top end, and it remains a single point of failure, so large systems eventually need replicas and partitioning.
Your service stores user sessions in memory. What's the problem?: Requests must stick to the same instance, load spreads unevenly, and a crash or deploy logs users out. Move sessions to a shared store (or use self-contained signed tokens) so any instance can serve any request.
You added 10x servers but throughput barely moved. Why?: The bottleneck is elsewhere: typically the database, a hot key, a shared lock, or a downstream dependency. Scaling the stateless tier cannot help work that is serialized behind one resource.

Trade-offs

Horizontal scaling trades simplicity for capacity and resilience. You gain a high ceiling and survivable failures, and you take on network partitions, partial failures, eventual consistency and operational complexity. The right design scales the parts that need it and keeps the rest simple.