A load balancer distributes incoming requests across a pool of servers so no single server is overloaded, and stops sending traffic to servers that are unhealthy. It is what lets you run ten identical app servers behind one address and add or remove servers without clients noticing. Examples: AWS ALB/NLB, Google Cloud Load Balancing, HAProxy, Nginx, Envoy.
In simple words
Like the person at a bank who points each customer to the next free counter, and stops sending people to a counter whose clerk has gone home. No single counter gets a huge queue while others sit idle.
Traffic is spread over healthy servers only; a failed health check takes a server out of rotation.
Why It Exists
Scale out: one server has a ceiling; many servers behind a balancer do not.
Availability: health checks remove failed servers automatically, so one crash is invisible to users.
Zero-downtime deploys: drain a server, update it, put it back.
How It Works
+------------------+
clients ---> | load balancer | -- health checks every few seconds
+------------------+
| | |
app1 app2 app3 (unhealthy -> removed from rotation)
Algorithms for picking a server:
Round robin: each server in turn. Simple; fine when requests cost about the same.
Weighted round robin: bigger servers get proportionally more.
Least connections / least outstanding requests: send to the least busy server. Better when request costs vary.
Hashing (IP hash, consistent hashing): the same key goes to the same server, useful for cache locality or sticky sessions.
Power of two choices: pick two servers at random and send to the less loaded. Cheap and surprisingly effective.
Types: Layer 4 vs Layer 7
Layer 4 (transport)
Layer 7 (application)
Sees
IPs and ports (TCP/UDP)
HTTP: path, host, headers, cookies
Routing
per connection
per request (/api vs /images, canary by header)
Speed
very fast, little processing
more CPU (parses HTTP, often terminates TLS)
Examples
AWS NLB, LVS
AWS ALB, Nginx, Envoy
When to Use It (and What to Watch)
Any tier with more than one instance: web servers, API services, even read replicas.
The balancer itself must be redundant (managed cloud LBs are; self-hosted ones run in pairs with a floating IP).
Sticky sessions work but hurt even spreading and failover; prefer stateless servers.
Key Pointers
Two jobs: spread load, and route around failures via health checks.
L4 is fast and protocol-agnostic; L7 understands HTTP and can route by content.
Least-outstanding-requests beats round robin when request cost varies.
Global load balancing (GSLB, DNS/anycast) picks a region; local load balancers pick a server within it.
Follow-up Questions
How does the load balancer know a server is unhealthy?
Active health checks (it calls /health every few seconds and removes a server after N failures) and often passive checks (it notices errors or timeouts on real traffic).
Why might round robin overload one server?
If some requests are much heavier (a report export vs a ping), a server can get several heavy ones in a row while others idle. Least-connections or least-outstanding-requests adapts to actual load.
What is connection draining?
When a server is removed (deploy, scale-in), the balancer stops sending it new requests but lets in-flight ones finish, so users don't see errors.