Latency, Throughput & Bandwidth

These three terms get used loosely in casual conversation but mean precisely different things in a system design interview, and mixing them up is an easy way to lose credibility. Knowing how they interact — not just their definitions — is what lets you reason correctly about API performance, video streaming, and database replication lag.

What Is Latency, Throughput & Bandwidth?

• Latency is how long it takes a single piece of data to get from A to B — the delay before anything happens.
• Bandwidth is the maximum amount of data a connection could carry per second, in theory — the size of the pipe.
• Throughput is how much data actually gets through per second in real conditions — usually less than bandwidth, because of overhead and congestion.

Where This Is Used

• Video calls on Zoom or FaceTime feel laggy on a high-latency connection even if the internet plan advertises fast "bandwidth" — the delay, not the pipe size, is the problem.
• Streaming services like Netflix care about throughput — how many megabits per second are actually arriving — to decide whether to drop your video quality.
• ISPs sell "bandwidth" (e.g. 500 Mbps plans), but your actual download speed (throughput) is usually lower once other devices, Wi-Fi interference, and network overhead are accounted for.
• Online gaming and video calls are latency-sensitive; a Google Drive backup is throughput-sensitive — the same network can be great for one and mediocre for the other.

A Day-to-Day Example

Think of a highway. Bandwidth is the number of lanes — how many cars could travel side by side at once. Throughput is how many cars actually get through per minute once you factor in traffic lights and congestion. Latency is how long it takes one single car to drive from one end of the highway to the other. A ten-lane highway (huge bandwidth) doesn't help a single car get there any faster if the drive itself takes an hour (high latency) — they're simply different things.

Deep Dive: Three Independent Axes, Not One Sliding Scale

Latency     = TIME for one unit of data/request to travel end-to-end
              e.g. "this API call takes 100ms"

Bandwidth   = MAX theoretical CAPACITY of a link
              e.g. "this link is rated for 1 Gbps"

Throughput  = ACTUAL data successfully moved per unit time, in practice
              e.g. "we're seeing 10,000 requests/sec" or "600 Mbps sustained"
              always <= bandwidth, reduced by overhead/congestion/retransmits

A satellite link: huge bandwidth (hundreds of Mbps) + terrible latency
                   (500-700ms round trip, due to physical distance to orbit)
                   -> proves bandwidth and latency are NOT the same axis.

The single most useful mental model: bandwidth is the width of the pipe, latency is how long it takes one drop of water to travel the length of the pipe, and throughput is how much water actually comes out the other end per second once you account for friction, leaks, and how full you're keeping the pipe.

Deep Dive: Percentiles — Why p99 Matters More Than Average

1,000 requests sorted by latency, fastest to slowest:

p50 (median): the 500th fastest request's latency
p95:          the 950th fastest request's latency (top 5% are slower)
p99:          the 990th fastest request's latency (top 1% are slower)

Example distribution:
p50 = 45ms      <- "typical" request, feels fast
p95 = 220ms     <- a meaningful chunk of users feel this
p99 = 800ms     <- ~1% of requests take 800ms or more (the "tail")
avg = 68ms      <- looks fine, HIDES the tail completely

Averages are misleading because a small number of very slow outliers get diluted across thousands of fast requests — a service can have a great-looking average latency while 1 in 100 users has a genuinely bad experience. That's precisely why SLOs are written in percentiles ("p99 latency under 500ms") rather than averages: p99 exposes the tail behavior that a small but real fraction of your users — or, at scale, a large absolute number of requests — actually experiences. If a page makes 20 sequential backend calls, even a 1% chance of hitting a slow p99 request per call means an ~18% chance that page load hits at least one slow call.

Deep Dive: Bandwidth-Delay Product and Why It Explains Slow Start

Bandwidth-Delay Product (BDP) = bandwidth × round-trip time (RTT)

Example: 100 Mbps link, 80ms RTT (e.g. cross-continent)
BDP = 100,000,000 bits/sec × 0.080 sec = 8,000,000 bits = 1 MB

This is how much data can be "in flight" (sent but not yet ACKed)
before the pipe is saturated. If TCP's congestion/receive window is
smaller than the BDP, the link sits IDLE waiting for ACKs — bandwidth
goes unused no matter how fast the link theoretically is.

This is the direct link back to TCP: a connection with a small congestion window relative to a large BDP path (high bandwidth, high latency — think transcontinental or satellite) can't saturate the link even though plenty of bandwidth is available, because it runs out of "in-flight" allowance and has to wait for ACKs before sending more. It's also why TCP's slow start takes measurably longer, in wall-clock time, on high-RTT paths — each doubling of cwnd only happens once per round trip, so a higher RTT means fewer doublings per second.

Key Pointers

• Latency is the time for a single unit of data (or a request) to travel from sender to receiver — measured in milliseconds, and dominated by propagation delay, queuing, and processing time.
• Bandwidth is the maximum theoretical capacity of a link — the width of the pipe, measured in bits per second (e.g. a 1 Gbps link).
• Throughput is the actual amount of data successfully transferred per unit time in practice — always less than or equal to bandwidth, reduced by congestion, retransmission, and protocol overhead.
• The bandwidth-delay product (bandwidth × round-trip latency) tells you how much data can be "in flight" at once — it's the theoretical amount of unacknowledged data a TCP connection needs to keep the pipe full.
• High bandwidth doesn't fix high latency, and low latency doesn't guarantee high throughput — they're independent axes, which is why a high-bandwidth satellite link can still feel "slow" for interactive use.
• p50/p95/p99 latency percentiles reveal tail behavior that an average hides; SLOs are typically defined against p95 or p99, not the mean.

How It Works

1. A request leaves the client; propagation delay (distance / speed of signal) plus per-hop processing and queuing delay at each router make up latency.
2. The link's bandwidth caps how much data can theoretically be pushed through per second.
3. Real throughput falls short of bandwidth due to protocol overhead (headers, ACKs), congestion control backing off under loss, and TCP's slow start needing time to ramp up to full speed.
4. On a high-latency, high-bandwidth link (e.g. transcontinental fiber), the bandwidth-delay product is large, meaning TCP needs a correspondingly large window to actually saturate the link — a small default window leaves bandwidth unused regardless of how fast the link theoretically is.
5. Aggregating raw latency numbers into percentiles (p50/p95/p99) at measurement time is what turns "average latency" into an SLO-worthy metric that actually reflects user experience.

Try It Yourself

# Measure raw round-trip latency to a host:
ping -c 20 example.com

# Break a request into its latency components (DNS, connect, TLS, TTFB, total):
curl -w '@-' -o /dev/null -s https://example.com <<'EOF'
   dns: %{time_namelookup}s\n
 conn: %{time_connect}s\n
  tls: %{time_appconnect}s\n
 ttfb: %{time_starttransfer}s\n
total: %{time_total}s\n
EOF

# See per-hop latency building up across the path (proxy for propagation + queuing):
mtr example.com                # live, continuous traceroute with per-hop stats

# Estimate real throughput on a transfer (compare to link's rated bandwidth):
curl -o /dev/null -s -w '%{speed_download} bytes/sec\n' https://example.com/largefile

# Compute bandwidth-delay product for a given link:
# BDP (bits) = bandwidth (bits/sec) * RTT (sec) — get RTT from `ping`,
# bandwidth from your ISP/link spec, then just multiply.

# In the browser: DevTools -> Network tab -> hover a request to see
# Queuing / DNS / Connecting / TLS / TTFB / Content Download breakdown.

Interview Q&A

Q: Why does adding more bandwidth not fix a slow cross-region API call?

A: Because propagation delay — the time for a signal to physically travel the distance, bounded by the speed of light in fiber — is a floor on latency that no amount of bandwidth changes. Bandwidth determines how much data can move per second once it's moving; it does nothing about how long the first bit takes to arrive. A cross-region call has a fixed minimum RTT set by geography, and only reducing hop count, moving compute closer to the user, or avoiding the round trip entirely (caching, async) actually helps.

Q: Why do we care about p99 latency instead of average latency?

A: Average latency is dominated by the bulk of fast requests and hides the tail — a service can report a great average while 1% of requests take 10x longer, and at scale that 1% is still a large absolute number of bad experiences, especially since a single page or workflow often makes many backend calls, compounding the chance of hitting a slow one. p99 (or p95) exposes that tail directly, which is why SLOs are written against percentiles, and why performance work often specifically targets tail latency rather than shaving the median.

Q: You have a link with plenty of bandwidth but a slow file transfer. How do you diagnose it?

A: First check whether it's actually a bandwidth ceiling (compare achieved throughput to the link's rated bandwidth) or a latency/window problem — if RTT is high and the achieved throughput is far below bandwidth, suspect the bandwidth-delay product: the TCP window in use may be too small to keep the pipe full across that RTT, in which case increasing the window size (or using parallel streams) fixes it without touching the link itself. If throughput is already near the link's bandwidth ceiling, then it genuinely is bandwidth-bound and no window tuning will help.

System Design Tie-in

In any "design a globally distributed service" interview, this trio directly justifies architectural choices: placing read replicas or CDN edges close to users reduces latency (propagation delay), which no amount of origin-server bandwidth can fix; sizing connection pools and worker counts is a throughput question, not a latency one; and defining your SLO in p95/p99 (not average) is the expected way to state a performance requirement, e.g. "p99 read latency under 200ms" rather than "average latency under 200ms," because the latter can be satisfied while a meaningful fraction of users have a bad experience.

Trade-offs

A common interview mistake is treating "we have plenty of bandwidth" as equivalent to "requests will be fast" — for latency-sensitive APIs (e.g. a synchronous chain of microservice calls), round-trip latency dominates user-perceived speed regardless of available bandwidth, which is why reducing hop count and using persistent connections matters more than upgrading link speed. Conversely, bulk data transfer (backups, replication, video downloads) is throughput-bound, so parallelizing streams or tuning TCP window sizes to match the bandwidth-delay product matters more than shaving milliseconds of latency. Interviewers often probe this with: "why does adding more bandwidth not fix a slow cross-region API call?" — the answer is that speed-of-light propagation delay is the floor on latency and no amount of bandwidth changes it.