A distributed transaction is one atomic unit of work that spans more than one database or resource manager: either every participant commits or none does. Two-phase commit (2PC) is the classic protocol that provides that all-or-nothing guarantee. Interviewers ask about it because it exposes the hard truth behind "just wrap it in a transaction": once the data lives in two places, atomicity needs a coordination protocol, that protocol has a blocking failure mode, and the cost of that failure mode is the reason modern microservice architectures usually avoid it and reach for the saga pattern instead. For a Backend or AI engineer, this is the topic that separates "I know ACID" from "I know what happens to ACID when the network gets involved".
Transfer money between two databases:
Coordinator DB-A (debit) DB-B (credit)
|--- COMMIT --------->| |
| | ok, committed |
|--- COMMIT ------------------------------X | (network drops / DB-B crashes)
| |
Result: money left A and never arrived at B. Each DB is internally ACID,
but the TRANSFER as a whole is not atomic.Each database can only guarantee atomicity for its own local transaction. If you simply send COMMIT to each in turn, there is a window between the first commit and the last where a failure leaves the system half-done, and a committed transaction cannot be undone by the database (there is no "un-commit"). The fix is to split the decision from the action: first make sure everybody is able to commit and has promised to, and only then tell everybody to do it. That ordering is the entire idea of 2PC.
Coordinator Participant 1 Participant 2
| | |
|=== PHASE 1: PREPARE (voting) ===================|
|--- PREPARE -------------->| |
|--- PREPARE ---------------------------------> |
| | write redo/undo | write redo/undo
| | to durable log, | to durable log,
| | keep locks | keep locks
|<-- VOTE YES --------------| |
|<-- VOTE YES ----------------------------------|
|
| all YES -> write COMMIT decision to its own durable log (the commit point)
| any NO / timeout -> write ABORT decision
|
|=== PHASE 2: COMMIT (or ABORT) ==================|
|--- COMMIT ---------------->| |
|--- COMMIT ----------------------------------> |
|<-- ACK -------------------| |
|<-- ACK ----------------------------------------|Two details carry all the weight. First, a participant that votes YES must have already made its work durable and must keep its locks: it has promised that it can commit no matter what, including after a crash and restart, so it has surrendered the right to abort on its own. Second, the coordinator's durable decision record is the real commit point of the whole distributed transaction. Once it is on disk, the outcome is fixed; phase 2 is just delivering a decision that already exists, and it can be retried until every participant acknowledges.
Failure What happens
----------------------------------------- ----------------------------------------------
Participant crashes BEFORE voting Coordinator times out, decides ABORT. Safe.
Participant crashes AFTER voting YES On restart it reads its log, sees "prepared",
and asks the coordinator for the outcome.
Coordinator crashes BEFORE deciding Participants that voted YES are "in doubt".
They cannot abort (they promised) and cannot
commit (they don't know the decision).
Coordinator crashes AFTER deciding Same for participants that never received it,
until the coordinator recovers and re-sends.
In-doubt participants keep holding row/table locks until the coordinator is back.This is why 2PC is called a blocking protocol. A prepared participant is stuck: committing might contradict a decision to abort, aborting might contradict a decision to commit. Participants can sometimes ask each other (a cooperative termination protocol): if any reachable participant already knows the outcome, or never voted YES, the rest can resolve; but if every reachable participant is merely prepared, they still have to wait. Meanwhile the locks they hold block other transactions, so one dead coordinator can stall unrelated traffic. Three-phase commit adds a pre-commit step to avoid blocking, but it only works under assumptions (bounded delays, no partitions) that real networks do not provide, so it is mostly a textbook topic; production systems that need non-blocking commit typically replicate the coordinator's state with a consensus protocol (Google Spanner, for example, runs 2PC across Paxos-replicated groups so that a participant or coordinator "failing" means a replica takes over).
+-------------------------------+
| Transaction Manager (TM) | e.g. a Java EE / JTA container
| = the 2PC coordinator |
+---------+-------------+-------+
XA | | XA
+-------v--+ +--v---------+
| Resource | | Resource | Resource Manager (RM) = a DB or
| Mgr: DB | | Mgr: queue | message broker that speaks XA
+----------+ +------------+
TM calls per RM: xa_start -> (work) -> xa_end -> xa_prepare -> xa_commit / xa_rollback
After a crash: xa_recover lists in-doubt (prepared) branches for the TM to resolveXA is the X/Open standard interface between a transaction manager and resource managers; in Java it surfaces as JTA and XA-capable data sources. It is 2PC with a vendor-neutral API, so a database and a message broker can commit atomically together. The SQL-level equivalents are engine specific: PostgreSQL has PREPARE TRANSACTION / COMMIT PREPARED (disabled by default because max_prepared_transactions is 0), and MySQL/InnoDB has XA START ... XA PREPARE ... XA COMMIT and XA RECOVER. A prepared transaction that nobody resolves keeps holding locks and, in PostgreSQL, can also hold back vacuum, so orphaned prepared transactions are a known operational hazard.
PREPARE TRANSACTION and MySQL's XA statements are the engine-level forms. Behaviour and defaults differ by engine.# Python 3 — a tiny in-memory 2PC simulation (real output below, from python3)
class Participant:
def __init__(self, name, can_commit=True):
self.name, self.can_commit = name, can_commit
self.state = "INIT" # INIT -> PREPARED -> COMMITTED / ABORTED
def prepare(self):
if self.can_commit:
self.state = "PREPARED" # promise: durable, can no longer unilaterally abort
return "YES"
self.state = "ABORTED"
return "NO"
def commit(self): self.state = "COMMITTED"
def abort(self): self.state = "ABORTED"
def two_phase_commit(participants, crash_after_prepare=False):
votes = [p.prepare() for p in participants] # phase 1
print(" votes:", dict(zip([p.name for p in participants], votes)))
if crash_after_prepare:
print(" COORDINATOR CRASHED before writing the decision")
return None
decision = "COMMIT" if all(v == "YES" for v in votes) else "ABORT"
print(" decision:", decision)
for p in participants: # phase 2
p.commit() if decision == "COMMIT" else p.abort()
return decision
def show(ps):
print(" states:", {p.name: p.state for p in ps})
print("Scenario 1: everyone votes YES")
ps = [Participant("orders"), Participant("payments"), Participant("inventory")]
two_phase_commit(ps); show(ps)
print("Scenario 2: payments votes NO")
ps = [Participant("orders"), Participant("payments", can_commit=False), Participant("inventory")]
two_phase_commit(ps); show(ps)
print("Scenario 3: coordinator crashes after collecting YES votes")
ps = [Participant("orders"), Participant("payments"), Participant("inventory")]
two_phase_commit(ps, crash_after_prepare=True); show(ps)
print(" -> PREPARED participants cannot commit or abort on their own; locks stay held")
# Output:
# Scenario 1: everyone votes YES
# votes: {'orders': 'YES', 'payments': 'YES', 'inventory': 'YES'}
# decision: COMMIT
# states: {'orders': 'COMMITTED', 'payments': 'COMMITTED', 'inventory': 'COMMITTED'}
# Scenario 2: payments votes NO
# votes: {'orders': 'YES', 'payments': 'NO', 'inventory': 'YES'}
# decision: ABORT
# states: {'orders': 'ABORTED', 'payments': 'ABORTED', 'inventory': 'ABORTED'}
# Scenario 3: coordinator crashes after collecting YES votes
# votes: {'orders': 'YES', 'payments': 'YES', 'inventory': 'YES'}
# COORDINATOR CRASHED before writing the decision
# states: {'orders': 'PREPARED', 'payments': 'PREPARED', 'inventory': 'PREPARED'}
# -> PREPARED participants cannot commit or abort on their own; locks stay held
-- PostgreSQL (not run locally; needs max_prepared_transactions > 0 in postgresql.conf)
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
PREPARE TRANSACTION 'transfer-42'; -- durable; session ends, locks remain held
-- later, from any session, after the coordinator has decided:
COMMIT PREPARED 'transfer-42'; -- or: ROLLBACK PREPARED 'transfer-42';
-- find orphans (an operational hazard):
SELECT gid, prepared, owner FROM pg_prepared_xacts;Q: Where should transaction boundaries be?
A: Around the smallest unit of work that must be atomic to preserve a business invariant, generally one use case in the service layer, inside a single database. Keep it short: do the reads and validation, write, commit. Do not hold a transaction open across slow external calls such as an HTTP request or a payment API, because the locks and the connection are held for the duration and one slow dependency can exhaust the pool. The trade-off is that a boundary that is too small lets invariants be violated between steps, while one that is too large hurts concurrency and invites deadlocks.
Q: How would you handle a distributed transaction?
A: First try to avoid needing one: redraw service boundaries so the invariant lives in a single database. If the data truly spans stores, there are two families. Use 2PC/XA when you control all participants, they support it, and you need strict atomicity, accepting blocking and lock holding. Otherwise use a saga (a sequence of local transactions with compensating actions) plus a transactional outbox for reliable messaging, accepting eventual consistency and the lack of isolation. The choice is really availability and coupling against strict atomicity.
Q: What happens if the coordinator fails during 2PC, and why is that a problem?
A: It depends on when. Before any participant has voted YES, participants can simply abort. After they have voted YES they are in doubt: they promised to commit if told to, so they cannot abort alone, and they do not know the decision, so they cannot commit alone. They hold locks until the coordinator recovers, which can stall other transactions touching the same rows. That is the blocking problem, and it is the core reason 2PC is a liability in systems that must stay available.
Q: Why is 2PC generally avoided across microservices?
A: Several costs compound. It holds locks across network round trips, so latency and contention rise with the number of participants. Availability drops because every participant and the coordinator must be up. It couples services to a shared protocol, and many stores and message brokers do not support XA at all, especially managed cloud services and third-party APIs. A saga trades those costs for eventual consistency: each service commits locally, and failures are handled by business-level compensation rather than by holding locks.
Q: What is XA?
A: XA is the X/Open standard for a transaction manager to drive 2PC across resource managers such as databases and message brokers, exposed in Java as JTA. It lets, for example, a database write and a message-queue send commit atomically. The cost is the same as 2PC: prepared branches hold locks, in-doubt transactions need recovery (xa_recover), and throughput suffers because of the extra durable log writes and round trips.
Follow-up: Does 3PC or consensus fix the blocking problem?
A: Three-phase commit adds a pre-commit state so participants can make progress after a coordinator failure, but it assumes bounded message delays and no partitions, so in a real network it can still produce inconsistent outcomes; it is rarely used. The practical fix is to make the coordinator itself fault tolerant by replicating its decision log with a consensus protocol like Paxos or Raft, so a new leader can finish the transaction. That removes single-coordinator blocking but keeps the latency and lock costs, and a transaction that spans a majority-less partition still cannot proceed.
Follow-up: A participant voted YES, then crashed and restarted. What must it do?
A: Because it logged the prepared state durably before voting, it recovers that state (including its locks) from the log, then asks the coordinator for the outcome and applies it. It must not release the locks or guess. The commit decision can be delivered more than once, so applying it must be idempotent.
Follow-up: A database write and a Kafka message must both happen. Do you use 2PC?
A: Usually not. The common answer is the transactional outbox: write the business row and an outbox row in one local database transaction, and let a separate relay publish the outbox row to the broker, with consumers deduplicating. That gives at-least-once delivery with no distributed commit, at the price of some publish delay and idempotent consumers.
In a design interview, the honest answer to "how do you keep the order, payment and inventory consistent" starts by asking whether they must be separate databases at all: a single database with a well-scoped local transaction is simpler and stronger than any distributed protocol. When they are separate services, sketch a saga with an outbox for reliable event publication, name the compensations, and call out that intermediate states are visible to readers. Reserve 2PC for cases where you control every participant and the participant count and latency budget are small, such as a database plus its own message broker in a single team's system, or inside a distributed database that runs 2PC over consensus-replicated shards (cross-shard transactions in systems like Spanner and other distributed SQL databases work roughly this way, with details varying by product). When sharding a relational database yourself, note that cross-shard transactions are exactly this problem, which is why shard keys are chosen so that most transactions stay on one shard.
The decision is between strict atomicity and availability plus autonomy. 2PC gives you all-or-nothing behaviour and, with the participants' locks held, isolation across the participants, so no reader sees a half-finished transaction. The cost is lock holding across network round trips, extra durable writes, tighter coupling, an availability ceiling set by the least reliable participant, and the in-doubt blocking window whenever the coordinator fails. It is the wrong choice across independently deployed services, across organisations, over third-party or non-XA resources, on high-latency links, or in any flow that must keep serving traffic when one component is down. It is the right choice when the participant set is small and under one team's control, the transaction is short, and inconsistency (money moving without arriving) is worse than temporary unavailability. If you cannot meet those conditions, prefer a single database, or a saga with compensations and an outbox.