← DBMS
Scaling & Distribution

Distributed Databases — Eventual Consistency & Quorums

Watch: "Distributed Systems 5.2: Quorums" — Martin Kleppmann

Once data lives on several machines, "what does a read return right after a write?" stops having a single answer. Eventual consistency, quorums and strong consistency are the three words interviewers use to probe whether you can reason about that question. They matter to a Backend / AI engineer because the same trade-offs appear when reading from a PostgreSQL replica, using DynamoDB or Cassandra, caching in Redis, or syncing a vector index from a source-of-truth table: in every case you must know whether a user can read their own write and what the system does when a node is slow or unreachable.

Deep Dive: Strong vs Eventual Consistency

STRONG (linearizable) read:               EVENTUAL read:

 t0  client writes x=1  -> ack             t0  client writes x=1 -> ack (after 1 replica)
 t1  ANY later read returns x=1            t1  read from a lagging replica -> x=0  (stale)
     (or something newer)                  t2  replication / repair catches up
                                           t3  read returns x=1
 cost: coordination on every op            guarantee: IF writes stop, all replicas
       (leader or quorum round trip)                  converge to the same value - eventually.
                                           NOT promised: how long, or what you read meanwhile.

"Eventual" is a promise about the end state, not about any particular read. That is why it is cheap: a write can be acknowledged after reaching one replica, and the rest are updated in the background. The price is a set of anomalies the application must tolerate or design around: stale reads, reading your own write from a different replica ("I saved it, why is it missing?"), and reads going backwards in time when consecutive requests hit different replicas. Milder guarantees between the extremes exist (read-your-writes, monotonic reads, causal consistency) and are often implemented by routing a user's reads to the same replica or by passing a version token.

Deep Dive: Quorums and the R + W > N Rule

N = 3 replicas of a key.  W = replicas that must ack a write.  R = replicas asked on a read.

  W=2, R=2  (R+W=4 > 3)          W=1, R=1  (R+W=2 <= 3)
  write lands on {1,2}            write lands on {1}
  read asks     {2,3}             read asks     {2}   or {3}
       overlap = {2}  -> the        no overlap -> read can return the OLD value
       read includes a node that
       has the new value

  Any W-set and any R-set out of N intersect when R + W > N  (pigeonhole).
  A "quorum" is a majority: floor(N/2) + 1   (N=3 -> 2, N=5 -> 3).

The reasoning is set overlap: if the nodes that acknowledged the write and the nodes consulted by the read must share at least one member, the read sees at least one copy of the latest value, and picks it using a version number or timestamp. Majority quorums for both give the largest fault tolerance for that guarantee: with N=5 and W=R=3 you can lose two nodes and still read and write. Tuning shifts cost around: W=N, R=1 makes reads cheap but a single dead node blocks writes; W=1, R=N does the opposite.

Important honesty for interviews: overlapping quorums are necessary but not sufficient for strong consistency. R + W > N gives a likely fresh read only with strict quorums, a fixed membership, and sane version ordering. It generally does not by itself give linearizability, because of concurrent writes racing with a read, partially failed writes that were not rolled back, last-write-wins decided by skewed clocks, and sloppy quorums (below).

Deep Dive: Sloppy Quorums, Read Repair and Convergence

Normal write (N=3 home replicas A,B,C; W=2)   Sloppy quorum during a partition:

 client -> coordinator -> A,B,C                 A is reachable, B and C are not.
 ack after 2 of {A,B,C} respond                 Coordinator writes to A + a stand-in node D
                                                (not one of the key's home replicas) and
Convergence mechanisms (leaderless stores):     acks W=2.  D holds a "hint" and later hands
 - read repair: a read that sees stale          the data to B or C (hinted handoff).
   replicas writes the newest value back        Availability up, but the W nodes are not
 - hinted handoff: stand-in delivers later      guaranteed to be the ones a strict read
 - anti-entropy: background comparison          quorum (A,B,C) will ask -> the R + W > N
   of replicas (e.g. Merkle trees)              overlap argument no longer holds.

Sloppy quorums are why "Dynamo-style stores are highly available" and "R + W > N guarantees fresh reads" cannot both be applied naively: the store chose to accept writes when the home replicas were unreachable, and it repairs later. That is the AP side of CAP in concrete mechanics. Conflicting concurrent writes to the same key are then resolved by last-write-wins (simple, can silently discard an update, sensitive to clock skew), by version vectors that surface both values as siblings for the application to merge, or by CRDT-style data types that merge automatically.

Deep Dive: Single-Leader Replication Is Also Eventually Consistent

-- PostgreSQL / MySQL, one primary + async read replica
Session A (primary):   INSERT INTO orders ...; COMMIT;        -- ack to user
Session B (replica):   SELECT ... FROM orders WHERE id = ?;    -- may not see it yet
                       (replication lag: WAL/binlog not yet applied on the replica)

-- Read-your-writes options:
--   1. read the user's own recent data from the primary
--   2. synchronous replication for that commit (costs latency, can block if the standby is down)
--   3. pass a log position / version token and wait until the replica has reached it

Eventual consistency is not exclusive to NoSQL. Any asynchronous replica is eventually consistent with its primary, and the same stale-read symptoms appear. The difference from leaderless quorum stores is who orders the writes: a single leader gives one ordering, so replicas differ only by how far behind they are, whereas leaderless designs allow concurrent writes on different replicas and need conflict resolution.

Key Pointers

How It Works

  1. A key is placed on N replicas (for example via consistent hashing); a coordinator node receives the client request.
  2. On a write, the coordinator sends the value with a version (timestamp or vector) to the replicas and waits for W acknowledgements before replying to the client.
  3. On a read, the coordinator queries R replicas, compares versions, and returns the newest one it saw.
  4. If the R responses disagree, read repair writes the newest value back to the stale replicas so the next read is more likely to be fresh.
  5. Replicas that missed a write (down, slow or partitioned) are brought up to date later by hinted handoff and by background anti-entropy that compares data between replicas.
  6. Concurrent writes to the same key are reconciled by the store's conflict-resolution rule; the application either accepts that rule or supplies its own merge logic.

Try It Yourself

# Python 3, no dependencies. Enumerates EVERY possible pair of (write set, read set)
# for N=3 to show exactly when a read can miss the latest write. Real output below.
import itertools

N = 3
REPLICAS = range(N)

def check(W, R):
    """Enumerate EVERY possible choice of W write-acks and R read-replicas.
    Write v1 (newer) lands on a W-subset; old value v0 remains on the rest.
    A read asks an R-subset and returns the highest-version answer."""
    stale = 0
    total = 0
    for wset in itertools.combinations(REPLICAS, W):
        for rset in itertools.combinations(REPLICAS, R):
            total += 1
            saw_new = any(r in wset for r in rset)
            if not saw_new:
                stale += 1
    return stale, total

print(f"N={N}")
print(f"{'W':>2} {'R':>2} {'R+W>N':>6} {'stale-read combos':>18}")
for W, R in [(1,1),(1,2),(2,1),(2,2),(1,3),(3,1)]:
    s, t = check(W, R)
    print(f"{W:>2} {R:>2} {str(R+W>N):>6} {f'{s} of {t}':>18}")

# Sloppy quorum: W=2 acks, but one ack came from a stand-in node D (hinted handoff),
# which is NOT one of the N home replicas {0,1,2}.
print()
print("Sloppy quorum: W=2 acked by {node0, stand-in D}; hint to node1/2 not yet delivered")
home_with_new = {0}                     # only node0 among the real replicas has v1
for rset in itertools.combinations(REPLICAS, 2):     # strict read quorum R=2 of home replicas
    print(f"  read from replicas {rset}: sees v1? {bool(set(rset) & home_with_new)}")

# Output:
N=3
 W  R  R+W>N  stale-read combos
 1  1  False             6 of 9
 1  2  False             3 of 9
 2  1  False             3 of 9
 2  2   True             0 of 9
 1  3   True             0 of 3
 3  1   True             0 of 3

Sloppy quorum: W=2 acked by {node0, stand-in D}; hint to node1/2 not yet delivered
  read from replicas (0, 1): sees v1? True
  read from replicas (0, 2): sees v1? True
  read from replicas (1, 2): sees v1? False
# Second script: W=1 write, R=2 reads, with read repair converging the replicas.
# N=3 replicas, W=1 write, R=2 read with read repair. Shows convergence.
replicas = [{"v": "v0", "ver": 0} for _ in range(3)]

def write_w1(target, value, ver):
    replicas[target] = {"v": value, "ver": ver}      # ack after ONE replica; others lag

def read_r2(idx_a, idx_b, repair=True):
    a, b = replicas[idx_a], replicas[idx_b]
    newest = a if a["ver"] >= b["ver"] else b
    if repair:
        replicas[idx_a] = dict(newest); replicas[idx_b] = dict(newest)
    return newest["v"]

def show(label):
    print(f"{label:<34}", [r["v"] for r in replicas])

show("initial")
write_w1(0, "v1", 1);                     show("after W=1 write (replica 0 only)")
print("read R=2 from (1,2), no repair ->", read_r2(1, 2, repair=False))
show("state unchanged")
print("read R=2 from (0,1) + repair   ->", read_r2(0, 1))
show("after read repair of (0,1)")
print("read R=2 from (1,2)            ->", read_r2(1, 2))
show("after read repair of (1,2)")
print("all replicas equal:", len({r['v'] for r in replicas}) == 1)

# Output:
initial                            ['v0', 'v0', 'v0']
after W=1 write (replica 0 only)   ['v1', 'v0', 'v0']
read R=2 from (1,2), no repair -> v0
state unchanged                    ['v1', 'v0', 'v0']
read R=2 from (0,1) + repair   -> v1
after read repair of (0,1)         ['v1', 'v1', 'v0']
read R=2 from (1,2)            -> v1
after read repair of (1,2)         ['v1', 'v1', 'v1']
all replicas equal: True
# cqlsh (Apache Cassandra) - not run here, shown for the syntax
# Consistency is chosen per request/session; replication factor is set on the keyspace.
CREATE KEYSPACE shop WITH replication = {'class':'SimpleStrategy','replication_factor':3};
CONSISTENCY QUORUM;                 -- cqlsh command: applies to following statements
-- With RF=3, QUORUM = 2 replicas. QUORUM writes + QUORUM reads overlap (2 + 2 > 3).
-- CONSISTENCY ONE would be faster but may read a stale replica.
-- Expected behaviour: with one replica down QUORUM still succeeds; with two down it fails.

Read the first table as a lesson in probability of staleness, not just a rule: with W=1, R=1 six of nine combinations miss the latest write, and the sloppy quorum example shows a "successful" W=2 write that one legal strict read (replicas 1 and 2) still does not see.

Interview Q&A

Q: What is eventual consistency?

A: It is a guarantee that if no new updates are made to an item, all replicas will eventually converge to the same value. It says nothing about how long that takes or what reads return in between, so clients can see stale data or see reads go backwards. The trade-off is that writes can be acknowledged early (low latency, high availability, survives partitions), at the cost of stale reads and conflict resolution, so it fits data like feeds and counters better than balances and stock.

Q: What is a quorum?

A: A quorum is the minimum number of replicas that must take part in an operation for it to count, usually a majority, floor(N/2) + 1. Choosing read and write quorums so that R + W > N makes them overlap, so a read contacts at least one replica that has the latest acknowledged write. The cost is latency and availability: every operation waits for several replicas and fails if a majority is unreachable, and the guarantee assumes strict quorums and sane versioning.

Q: Strong vs eventual consistency?

A: Strong consistency makes the replicated store behave like a single copy: once a write is acknowledged, every subsequent read reflects it, which needs coordination (a leader, a consensus round or an overlapping quorum) on the request path. Eventual consistency drops that coordination so operations are faster and stay available during failures, but reads may be stale and concurrent writes need reconciliation. Pick strong for invariants that cross records (money, inventory, uniqueness) and eventual where slightly old or merged data is harmless.

Follow-up: Does R + W > N guarantee linearizability?

A: No, generally not on its own. A read concurrent with a write can see the new value on one replica and the old on another and different readers may then disagree; a write that fails after reaching fewer than W replicas is usually not rolled back, so a later read may or may not return it; last-write-wins with skewed clocks can discard the newest write; and sloppy quorums break the overlap. Stronger guarantees need extra mechanisms such as synchronous read repair before returning, or a consensus protocol.

Follow-up: What is a sloppy quorum and hinted handoff?

A: When some of a key's home replicas are unreachable, a sloppy quorum lets the coordinator count acknowledgements from other healthy nodes, which store the data with a hint and forward it once the home replica returns (hinted handoff). Writes keep succeeding during failures, but a subsequent strict read quorum may not include the stand-in, so the overlap argument fails until the hints are delivered. It is a deliberate availability-over-consistency choice.

Follow-up: How do you resolve conflicting writes in an eventually consistent store?

A: Last-write-wins picks the highest timestamp: cheap and convergent, but it silently discards the losing write and depends on clock quality. Version vectors detect true concurrency and keep both values as siblings for the application to merge, which is correct but pushes complexity to the app. CRDTs (counters, sets, registers with merge semantics) make merges automatic for specific data shapes but cannot express arbitrary invariants such as "balance never below zero".

System Design Tie-in

In a design discussion, state consistency needs per data type and then choose mechanisms. A shopping cart or a "recently viewed" list tolerates eventual consistency with merge-on-read (the classic Dynamo use case), so a leaderless store with W=1 or W=2 keeps the site writable during partial failures. A payment ledger or inventory decrement needs a single ordering: use one leader per partition or a consensus-based store, and make a partitioned minority return errors. Read-heavy systems with PostgreSQL read replicas need an explicit read-your-writes plan (recent-write reads from the primary, or session-pinned replicas), otherwise "post a comment then refresh" appears broken. Quorum settings are also an operations lever: raising R and W to QUORUM in a five-node cluster tolerates two failures for both reads and writes, and tuning consistency down for analytics reads keeps them from blocking interactive traffic. In AI systems, a vector index or cache fed asynchronously from the source-of-truth table is an eventually consistent view; delete and permission changes must therefore be applied with a tombstone or filter checked against the authoritative store, not only trusted from the index.

Trade-offs

Eventual consistency buys latency, write availability during failures, and horizontal scale by removing coordination from the hot path; it costs stale reads, anomalies such as reads going backwards, and conflict-resolution logic that lives in your application. Quorums are a dial between the two, not a switch: raising R and W improves the chance of fresh reads and increases latency and the risk of failing operations when replicas are slow, while the R + W > N guarantee itself is weakened by sloppy quorums, concurrent writes and clock skew. It is the wrong choice when correctness depends on a single global order (money, stock, unique constraints), where a leader-based or consensus-based store with transactions is simpler and safer, and it is overkill when one well-provisioned primary with replicas already meets your scale, because the operational complexity of a leaderless cluster is real.