Foundations

Availability & Reliability — Nines, Redundancy, Failover & SLOs

Availability is the fraction of time a system is up and answering correctly. Reliability is the broader promise that it keeps doing the right thing over time, including not losing data. Both are designed in, mostly through redundancy (more than one of everything that matters) and failover (switching to the spare quickly and safely when the primary fails).

In simple words

A hospital keeps a backup generator because the power will fail at some point; the question is only when. Availability is about having that spare ready and switching to it fast, so patients barely notice.

Two of everything, in different failure domains. If the primary dies, the standby is promoted and health checks route around the dead node.

Deep Dive: The Nines

Availability	Downtime per year	Per month	Per day
99% ("two nines")	~3.65 days	~7.3 h	~14.4 min
99.9%	~8.8 h	~44 min	~1.4 min
99.99%	~53 min	~4.4 min	~8.6 s
99.999%	~5.3 min	~26 s	~0.9 s

Each extra nine costs roughly an order of magnitude more effort: redundant regions, automated failover, careful deploys. Pick the target from what an outage costs the business, not from what sounds impressive.

Deep Dive: Availability in Series and in Parallel

In series (a request needs ALL of them):
  LB 99.99% -> App 99.9% -> DB 99.9%
  total = 0.9999 x 0.999 x 0.999 ~ 99.79%   (worse than any single part)

In parallel (a request needs ANY one of them):
  two independent app servers, each 99%
  total = 1 - (0.01 x 0.01) = 99.99%       (redundancy multiplies nines)

This is why every hop you add to a synchronous path lowers availability, and why redundancy at each tier matters. The parallel formula assumes independent failures; replicas in the same rack, zone or region share failure causes (power, network, a bad deploy), which is why redundancy is spread across availability zones and sometimes regions.

Deep Dive: SLI, SLO, SLA and Error Budgets

SLI (indicator): a measured number, such as the percentage of requests served successfully within 300 ms.
SLO (objective): the internal target for that SLI, such as 99.9% over 30 days.
SLA (agreement): a contract with customers, usually looser than the SLO, with penalties if missed.
Error budget: 100% − SLO. At 99.9%, you may "spend" ~43 minutes of failure a month. When the budget is burning fast, teams slow down risky deploys; when there's plenty left, they ship faster.

Deep Dive: Failover Patterns

Active-passive: primary serves, standby replicates and waits.
  + simple, no write conflicts     - standby idle; failover takes seconds-minutes;
                                     async replication may lose the last writes

Active-active:  all nodes serve traffic.
  + no idle capacity, instant      - concurrent writes to the same data need
    failover for stateless tiers     conflict handling (or partitioned ownership)

Failure detection: heartbeats + timeouts. Too short -> false failovers;
too long -> longer outages. Split brain (two primaries) is prevented with
a quorum / consensus (e.g. a lease held via etcd/ZooKeeper) or fencing.

Key Pointers

No single point of failure on the critical path: at least two of every component, in different failure domains.
Series dependencies multiply unavailability; remove hops or make them asynchronous.
Graceful degradation beats total failure: serve stale cache, hide a non-essential widget, queue writes for later.
Durability is separate from availability: replicate data and take backups you have actually restored from.
Most outages come from changes (deploys, config). Canary releases and fast rollback are availability features.

How It Works

Define SLIs and an SLO for the user-facing path.
Make stateless tiers redundant across zones behind a health-checking load balancer.
Replicate the data tier with automatic, fenced failover; decide how much data loss (RPO) and downtime (RTO) is acceptable.
Add timeouts, retries with backoff, and circuit breakers around dependencies.
Plan degraded modes and test failure deliberately (game days, chaos testing).

Follow-up Questions

What's the difference between RPO and RTO?: RPO (recovery point objective) is how much data you can afford to lose, measured in time: "at most 5 seconds of writes". RTO (recovery time objective) is how long recovery may take: "back within 15 minutes". Synchronous replication lowers RPO at a latency cost; automated failover lowers RTO.
Your system has 5 services in a synchronous chain, each 99.9%. What's the availability?: About 0.999⁵ ≈ 99.5%, roughly 3.6 hours of downtime a month. Reduce the chain, make some calls asynchronous, or add fallbacks.
How do you avoid split brain during failover?: Only a node holding a lease or quorum majority may act as primary, and the old primary is fenced (its writes rejected, for example with an epoch/term number checked by storage) so it can't keep accepting writes after it's been replaced.

Trade-offs

Higher availability costs money (duplicate capacity), complexity (failover logic, multi-region data) and sometimes consistency: during a network partition, staying available can mean serving or accepting data that isn't up to date (see CAP). Match the target to the business need per feature, not one number for the whole system.