A saga is a way to get a business-level "all or nothing" across several services without a distributed lock: the work is split into a sequence of local transactions, each committed in its own database, and if a later step fails, earlier steps are undone by running compensating actions. The idea comes from a 1987 paper on long-lived transactions, and it is now the default answer to "how do you handle a distributed transaction across microservices". Interviewers probe it because the honest answer is a trade: you give up isolation and instant atomicity to keep availability and service autonomy, and you take on failure handling, idempotency and observability work that a single ACID transaction used to do for free. For a Backend or AI engineer building order, payment or agent-workflow pipelines, this is the pattern behind most multi-step, multi-service flows. It is the alternative to two-phase commit.
Order saga (each step is a LOCAL ACID transaction in one service)
T1 create_order T2 reserve_stock T3 charge_card T4 ship
| | | |
C1 cancel_order C2 release_stock C3 refund (none: pivot/retriable)
Happy path: T1 -> T2 -> T3 -> T4 saga COMPLETED
Failure at T3: T1 -> T2 -> T3 (declined) -> C2 -> C1 saga ABORTED (compensated in reverse)Each local transaction commits and becomes visible immediately, so a saga cannot "roll back" in the database sense. A committed change can only be countered by a new business action that semantically undoes it: release the stock, cancel the order, issue a refund. Compensation is therefore a real piece of design, not an automatic feature, and it need not restore the exact prior state (a refund leaves a charge and a refund on the statement; a cancelled order still exists with status CANCELLED). Compensations run in reverse order and must themselves be retried until they succeed, because there is nothing to fall back to if a compensation fails.
CHOREOGRAPHY (events, no central brain) ORCHESTRATION (a coordinator drives it)
Orders --OrderCreated--> Inventory +-------------------+
Inventory --StockReserved--> Payments | Saga Orchestrator | holds state:
Payments --PaymentFailed--> Inventory | step, status, log | step 2 of 4
Inventory --StockReleased--> Orders +--+-------+-------+-+
cmd| cmd| cmd|
Each service listens and reacts. The flow is v v v
implicit in who subscribes to what. Orders Inventory PaymentsIn choreography, services publish events and react to each other's events; there is no single owner of the flow, so it is loosely coupled and has no central component to fail, but the overall process exists only as an emergent property of subscriptions. As steps grow, it gets hard to see the whole flow, to debug where an order is stuck, and to avoid cyclic dependencies. In orchestration, a dedicated coordinator sends commands to participants and tracks the saga state (which step, forward or compensating), so the flow is explicit, testable and observable, at the cost of a component that must be durable and highly available and a risk that it accumulates business logic. Workflow engines such as Temporal, AWS Step Functions, Camunda and Netflix Conductor exist mainly to be that orchestrator with durable state and retries. A common rule of thumb: choreography for two or three steps, orchestration once the flow has branching, timeouts or many participants.
Orchestrator sends "reserve 3 of sku-1 for c-100"
-> Inventory commits, then its reply is lost
-> Orchestrator retries the same command
Without a guard: stock is reserved twice. With a guard: second delivery is a no-op.
-- SQLite (portable idea: claim the command id and apply the effect in ONE local transaction)
CREATE TABLE stock (sku TEXT PRIMARY KEY, available INT NOT NULL CHECK (available >= 0));
CREATE TABLE processed_commands (command_id TEXT PRIMARY KEY);
INSERT INTO stock VALUES ('sku-1', 10);
INSERT OR IGNORE INTO processed_commands VALUES ('c-100');
UPDATE stock SET available = available - 3 WHERE sku='sku-1' AND changes() = 1;
SELECT 'after 1st delivery', available FROM stock;
INSERT OR IGNORE INTO processed_commands VALUES ('c-100');
UPDATE stock SET available = available - 3 WHERE sku='sku-1' AND changes() = 1;
SELECT 'after redelivery', available FROM stock;
-- after 1st delivery|7
-- after redelivery|7Networks and brokers give you at-least-once delivery at best, and exactly-once processing is something you build on top with idempotency. The pattern is: every command carries a unique id, and the participant records that id in the same local transaction as the business effect, relying on a primary-key or unique constraint to reject duplicates. The changes() = 1 guard is SQLite-specific shorthand for "only apply the update if the claim insert actually inserted a row"; in application code you would check the rows-affected count, or use INSERT ... ON CONFLICT DO NOTHING in PostgreSQL. Compensations need the same treatment, since "release stock" delivered twice must not release twice. Publishing the next event reliably from that same transaction is what the transactional outbox pattern is for.
Time Saga 1 (order A) Another reader / saga
t1 T2 reserve_stock (commit) --> sees stock = 7 (a saga-owned intermediate state)
t2 T3 charge_card (declined)
t3 C2 release_stock (commit) --> stock back to 10
Reader at t1 acted on a value that was later undone.
ACID: Isolation is the "I". A saga gives A, C (per local step) and D, but NOT I.
Result: dirty reads, lost updates and non-repeatable reads across the whole saga
are possible, because each step's effects are visible as soon as it commits.A saga is ACD without the I: atomicity is approximated by compensation, and there is no isolation between concurrent sagas or between a saga and other readers. Standard countermeasures include a semantic lock (mark the order as PENDING so other flows treat it as in progress), commutative updates (design operations that can be applied in any order), a pessimistic view (reorder steps so the riskiest, least compensable step comes early), re-reading a value before overwriting it, and version numbers or optimistic checks to detect that data changed underneath the saga. Structure the steps as compensable steps, then one pivot step (the point of no return, after which the saga will complete), then retriable steps that must eventually succeed. None of this is free, and it is the honest cost you must state when recommending a saga.
PENDING.CONFIRMED) and emits a completion event.CANCELLED.# Python 3 — a saga with compensations and an idempotency guard (real output below, from python3)
class PaymentDeclined(Exception):
pass
log = [] # what actually happened, in order
processed = set() # idempotency keys already applied
def once(key, fn):
"""Idempotent step: applying the same key twice has no extra effect."""
if key in processed:
log.append(f"skip duplicate {key}")
return
processed.add(key)
fn()
def create_order(o): log.append("order CREATED")
def cancel_order(o): log.append("order CANCELLED (compensation)")
def reserve_stock(o): log.append("stock RESERVED")
def release_stock(o): log.append("stock RELEASED (compensation)")
def charge_card(o, fail):
if fail:
log.append("payment DECLINED")
raise PaymentDeclined()
log.append("payment CHARGED")
def run_saga(order_id, decline=False):
log.clear(); processed.clear()
# each step: (name, action, compensation)
steps = [
("create_order", lambda: create_order(order_id), lambda: cancel_order(order_id)),
("reserve_stock", lambda: reserve_stock(order_id), lambda: release_stock(order_id)),
("charge_card", lambda: charge_card(order_id, decline), None),
]
done = []
try:
for name, action, comp in steps:
once(f"{order_id}:{name}", action)
done.append((name, comp))
except PaymentDeclined:
for name, comp in reversed(done): # undo in reverse order
if comp:
once(f"{order_id}:undo:{name}", comp)
log.append("SAGA ABORTED")
return
log.append("SAGA COMPLETED")
print("Happy path:")
run_saga("A1"); print("\n".join(" " + l for l in log))
print("Payment declined:")
run_saga("A2", decline=True); print("\n".join(" " + l for l in log))
print("Duplicate delivery of the same command (idempotency key):")
log.clear(); processed.clear()
once("A3:reserve_stock", lambda: reserve_stock("A3"))
once("A3:reserve_stock", lambda: reserve_stock("A3"))
print("\n".join(" " + l for l in log))
# Output:
# Happy path:
# order CREATED
# stock RESERVED
# payment CHARGED
# SAGA COMPLETED
# Payment declined:
# order CREATED
# stock RESERVED
# payment DECLINED
# stock RELEASED (compensation)
# order CANCELLED (compensation)
# SAGA ABORTED
# Duplicate delivery of the same command (idempotency key):
# stock RESERVED
# skip duplicate A3:reserve_stockQ: How would you handle a distributed transaction?
A: From the saga angle: break the business transaction into local transactions per service, give each a compensating action, coordinate them by orchestration or choreography, and make every step idempotent with reliable messaging via an outbox. On failure, run compensations in reverse. The trade-off is that you get eventual consistency and no isolation, so you have to design for visible intermediate states; in return you avoid cross-service locks and 2PC's blocking, which keeps services available and independently deployable.
Q: Orchestration vs choreography: which would you pick?
A: It depends on the size and stability of the flow. Choreography is simple and fully decentralised for two or three participants, but the process becomes implicit and hard to trace as it grows, and cycles between services creep in. Orchestration makes the flow explicit, easy to test and monitor, and puts timeouts and retries in one place, at the price of a component that must be durable and that can attract business logic. For non-trivial workflows I would generally choose orchestration, ideally on an existing workflow engine rather than a hand-rolled one.
Q: What is a compensating action, and what are its limits?
A: It is a business-level operation that semantically undoes a committed step: release reserved stock, cancel an order, refund a payment. It is not a rollback: the original effect was already visible and the compensation is a new transaction that can itself fail and must be retried. Some effects cannot be undone (an email or push notification already sent, a physical shipment), so those steps should be placed late, after the point where failure is unlikely, or replaced with something reversible such as a "hold" that is later captured.
Q: Why does idempotency matter so much in sagas?
A: Messages are delivered at least once, and a caller cannot tell "the request was lost" from "the reply was lost", so retries and duplicates are normal. If a step is not idempotent, a retry double-charges or double-reserves. The usual fix is a unique command id checked in the same local transaction as the effect, backed by a unique constraint; compensations need it too. The trade-off is a dedup table that has to be retained long enough to cover the retry window and then cleaned up.
Q: A saga has no isolation. What goes wrong and how do you cope?
A: Because each local commit is immediately visible, another request can read or overwrite data that a still-running saga will later compensate, giving dirty reads and lost updates at the business level. Coping techniques are semantic locks (a PENDING state other flows respect), commutative operations, re-reading before writing, version checks, and ordering steps so the riskiest come first. If the business cannot tolerate visible intermediate states at all, that is a signal the invariant belongs inside one database, not in a saga.
Follow-up: What if the orchestrator crashes mid-saga?
A: Its state must be persisted (a database row or a workflow engine's event history), so on restart it reloads the saga and resumes from the last recorded step or compensation. That only works if each step is idempotent, since the orchestrator may not know whether the last command was applied before the crash. The orchestrator is then a stateful component you have to run with high availability, which is the main operational cost of orchestration.
Follow-up: What if a compensation itself keeps failing?
A: Compensations should be designed to be retriable and to eventually succeed, so retry with backoff. If they still fail, the saga is stuck in a known state and should be surfaced to an operator or a dead-letter queue for manual repair, with alerts. Forward-only recovery (retrying the original step instead of undoing) is sometimes better once you are past the pivot step. There is no automatic rollback to lean on, which is exactly why compensations are kept simple.
Follow-up: How do you publish an event and update the database atomically?
A: Use the transactional outbox: write the business row and an outbox row in one local transaction, and have a separate relay (a poller or change-data-capture) publish outbox rows to the broker. Consumers deduplicate, because the relay can publish a row more than once. This avoids a distributed transaction between database and broker, at the cost of some publish latency and at-least-once semantics.
Any checkout, booking or onboarding flow that touches several services is a saga whether or not you call it one, so in a design interview state the steps, the compensation for each, and where the pivot is: for an order, create as PENDING, reserve stock, authorise (not capture) payment, then confirm and capture, with release and void as compensations. Say which parts are orchestrated (a workflow engine holding state and timers) and which are reactive events, and mention the outbox so that each service's commit and its message stay consistent. Then address what users and other services see in the meantime: pending states, idempotency keys on client requests so a double-clicked "pay" does not double charge, and dashboards or alerts for sagas stuck for too long. The same shape applies to AI pipelines, where an ingestion or agent workflow calls several systems (an object store, a vector index, a metadata database) and needs defined behaviour when the third step fails, such as removing the partially written embeddings.
Choosing a saga means choosing availability and service autonomy over strict atomicity and isolation. You avoid distributed locks and the in-doubt blocking of two-phase commit, and each service keeps its own database and its own failure domain. The cost is that correctness moves into application code: compensations to write and test, idempotency everywhere, an outbox, saga-state storage, timeouts, monitoring for stuck flows, and a user-visible window of intermediate states with anomalies that ACID would have prevented. It is the wrong choice when the invariant is strict and must never be transiently violated (move it into one database), when compensation is impossible or unacceptable, when there are only two tightly owned participants where a simple local transaction or an XA setup is acceptable, or when the team lacks the operational maturity to run and debug asynchronous workflows. Do not adopt a saga just because the system was split into microservices; first ask whether the split created the problem.