Networks fail, clients time out, brokers redeliver and workers crash halfway through a job, so every serious backend eventually executes the same logical request more than once. An operation is idempotent when applying it many times has the same effect as applying it once, and idempotency is the tool that turns "we retry on failure" from a source of double charges into a safe default. Interviewers probe it because it separates people who quote "exactly-once" from people who know it is really at-least-once delivery plus idempotent processing, and because the follow-up is always a concrete race: what happens when two retries of the same request arrive at the same moment? For a Backend or AI engineer it applies to payment APIs, webhook receivers, queue consumers, and expensive LLM or inference jobs where a duplicate means both a wrong side effect and a real bill.
Idempotency means that doing something once has the exact same result as doing it five times. Pressing an elevator call button is idempotent: pressing it once, or pressing it ten times out of impatience, still just makes the elevator come once. That property matters online because computers retry things constantly, since a slow or dropped internet connection makes an app unsure whether a click actually went through, so it often tries again & if the action isn't idempotent, that retry can cause it to happen twice.
"Exactly-once" is the promise that an action, like charging a credit card or sending a notification, happens precisely one time, never zero and never two or more. It sounds simple, but it's genuinely hard to guarantee over an unreliable network, because a computer sending a request often can't tell the difference between "that never arrived" and "that arrived and worked, but I never heard back." Systems get as close to exactly-once as possible by combining retries (to make sure something happens at least once) with idempotency (to make sure it never happens more than once).
Imagine mailing a birthday card. If you're not sure your first letter arrived, you might mail a second copy just in case & a normal letter is not idempotent, since now your friend gets two cards and might think you forgot you already sent one. But if instead you wrote "birthday card for June 2026" on the envelope, and your friend's mail carrier were smart enough to simply not deliver a second envelope with the same label, it wouldn't matter how many copies you sent: your friend would still end up with exactly one card. That label is exactly what an idempotency key does for a computer system.
The unavoidable ambiguity (producer P, consumer C, network in between):
P --- msg ---> C C processes it, sends ack
P <--- ack --- X ack is lost, or C crashed AFTER the side effect but BEFORE the ack
P: "did C process it?" P cannot tell "never arrived" from "processed, ack lost".
P's only two choices:
do not resend -> AT-MOST-ONCE : may lose the message (fire and forget, ack before processing)
resend -> AT-LEAST-ONCE : may duplicate it (ack after processing, retry until acked)
EXACTLY-ONCE delivery over an unreliable network is not on the menu.
What systems actually build: at-least-once delivery + idempotent processing = "effectively once"The three named semantics differ only in when the receiver acknowledges relative to doing the work. Acknowledge first and crash, and the work is lost (at-most-once). Do the work first and crash before acknowledging, and the sender retries, so the work happens again (at-least-once). Closing that gap needs the sender and receiver to agree on a single atomic fact ("this message has been applied") across a lossy link, which is the same family of impossibility as the two-generals problem. So a claim of exactly-once delivery should always prompt the question "what exactly is being counted, and where does the boundary end?". What is achievable is exactly-once effect: let the message arrive several times, but make processing it a second time a no-op.
This is why most production designs pick at-least-once and push the responsibility to the consumer. At-most-once is only acceptable when loss is cheap (metrics samples, best-effort notifications). The rest of this page is about how to make the consumer side safe: idempotency keys, natural idempotency, dedup tables, and what happens when duplicates race each other.
Client Server
key = uuid4() (generated ONCE per logical operation, reused on every retry)
POST /charges Idempotency-Key: K body B1
1. claim K (INSERT ... key=K, req_hash=H(B1), status=in_progress)
2. do the work, then store response R, status=done
<-- timeout / connection reset -- (client never saw R)
POST /charges Idempotency-Key: K body B1 (retry)
3. K exists, H(B1) matches, status=done -> return stored R, do NOT redo work
POST /charges Idempotency-Key: K body B2 (bug: same key, different body)
4. K exists, hash differs -> reject (commonly 422 / 409), never guess
Row: key (PRIMARY KEY, scoped per tenant/user/endpoint) | req_hash | status | response | created_at (for TTL)Three design decisions matter. First, the key is generated by the client and reused across retries; a key the server invents per attempt is useless, because the whole problem is that the client does not know whether the first attempt landed. Second, the result is stored with the key, so a retry gets the original answer (same charge id, same status code) rather than "already processed". Third, the request hash guards against misuse: reusing a key with a different payload is almost certainly a client bug, and silently returning the first response would hide it. Payment APIs such as Stripe popularised the Idempotency-Key header; an IETF draft exists for standardising it, so check current docs for exact status codes and retention rules of any provider you name.
Keys cannot be kept forever, so they carry a TTL (payment providers commonly keep them for on the order of a day; treat that as "check the provider's docs"). The TTL must exceed the longest realistic retry window, including a client that retries from an offline queue days later. After expiry a very late retry is processed as new, so for money you pair the key with a natural business constraint (for example a unique order id) as a second line of defence. Also scope keys per tenant or user and per endpoint, or two unrelated clients that both send K=1 collide and one silently gets someone else's response, which is a security bug, not just a correctness bug.
Naturally idempotent (safe to repeat) NOT idempotent (repeat changes the world)
PUT /users/7 {name:"Asha"} POST /charges {amount:500}
SET balance = 100 UPDATE balance = balance + 100
DELETE /orders/42 (second call: 404/204) INSERT INTO ledger(...) (no natural key)
INSERT ... ON CONFLICT (order_id) DO NOTHING send_email(), call LLM, enqueue job
UPDATE ... WHERE version = 7 (compare-and-set)
Make a non-idempotent op idempotent, cheapest first:
1. Reshape it: "set to X" instead of "add X"; client supplies the id (PUT /orders/{client_id}).
2. Business unique constraint: UNIQUE(order_id) on the charges table.
3. Dedup table: in ONE transaction INSERT processed(message_id) + apply effect. ("inbox" pattern)
4. Idempotency-key table with stored response (the API-facing version of 3).Prefer designs where the operation is naturally idempotent, because they have no extra state to expire, race or lose. Where that is impossible (a charge is inherently "do something new"), you need memory of what has been done, and the correctness condition is that the dedup record and the side effect commit atomically. If both live in the same database this is one transaction: either the effect and the "seen" mark are both there or neither is. If you record "seen" first and then crash, the retry is wrongly skipped and the work is lost; if you do the work first and record afterwards, a crash in between causes a duplicate. That single ordering argument is what interviewers are usually fishing for.
The difficulty appears when the side effect is outside your database: an email, a payment provider, an LLM call. Now no local transaction can cover both. The standard answer is to make the downstream call idempotent too, by passing your key along (many payment providers accept one), and to write the intent durably first, then execute, using an outbox and a relay so a crash re-drives the step rather than losing it. For multi-step business flows that spans several services, see Saga Pattern (each step and each compensation must itself be idempotent) and Change Data Capture (the log-based way to publish the outbox; it also delivers at-least-once, so consumers still dedupe).
WRONG: check, then act RIGHT: claim first, using a UNIQUE constraint
T1: SELECT key=K -> not found T1: INSERT key=K (in_progress) -> ok, I own it
T2: SELECT key=K -> not found (same moment) T2: INSERT key=K -> UNIQUE violation
T1: charge, INSERT key=K T2: SELECT key=K -> status=in_progress
T2: charge, INSERT key=K -> reply 409 "retry shortly" (or wait and replay)
--> two charges. Nothing in the app T1: charge, UPDATE key=K done, response=R (one txn)
is "wrong"; the gap is the bug. T2's next retry: status=done -> return R
Crash hole: T1 claims K and dies before finishing -> K stuck "in_progress" forever.
Fix: store a lease/started_at; treat an in_progress row older than the max job time as abandoned
(retry may re-claim it), or do claim + effect + "done" in ONE transaction when the effect is in the same DB.Any check-then-insert sequence has a window between the check and the write, and two retries (a client timeout racing the original request, a load balancer retry, two consumers after a rebalance) fit inside it easily. The fix is not a smarter if: it is to make the database arbitrate. A PRIMARY KEY or UNIQUE constraint is an atomic test-and-set, so exactly one inserter wins and the others get an error that tells them precisely "someone else owns this key". The loser then chooses to reject with a retryable status, block briefly and replay the stored response, or, for a slow job, return the job handle. In Redis the same idea is SET key value NX EX ttl; a lock or claim with a TTL always risks the owner outliving its lease, so the downstream effect should still be idempotent as a backstop.
The remaining hole is the crash between claim and completion. If the claim and the effect commit in the same transaction there is nothing half-finished to recover. If the effect is external, the row must record enough (a lease timestamp, plus the downstream idempotency key) that a later attempt can safely resume: re-issuing the downstream call with the same key either does the work once or returns the earlier result.
Kafka idempotent producer (enable.idempotence=true; the default in recent client versions, check yours)
producer gets a ProducerId; each (producer, partition) send carries a sequence number
broker: seq already written -> drop the duplicate; out-of-order gap -> error
covers: retries of a send within the producer's lifetime, to one partition
does NOT cover: a restarted producer with a new id, or your consumer's external side effects
Kafka transactions (transactional.id, read_committed consumers)
atomically write to several partitions AND commit the consumer offsets -> consume-transform-produce
loop is "exactly-once" INSIDE Kafka. The moment the handler also writes to Postgres,
calls Stripe or an LLM, that effect is outside the transaction: you are back to at-least-once + idempotency.
LLM / payment call with a timeout:
attempt 1 -> provider ran it, response lost -> retry = second charge / second billed generation
guard: your own idempotency key + stored result; pass the provider's idempotency header if it has oneKafka's marketing phrase "exactly-once semantics" is accurate only within its scope: the idempotent producer removes duplicates caused by producer retries, and transactions make a read-process-write cycle across Kafka topics atomic together with offset commits. Both are worth naming in an interview, hedged as "depending on version and configuration, and only for data that stays inside Kafka". A consumer that inserts into a database still needs its own dedup (for example storing the topic-partition-offset or a business id with the write, in one transaction). More on brokers and their guarantees in Message Queues & Event Streaming.
Payment and LLM retries are the same problem with different stakes. A timeout on a charge call does not tell you whether the charge happened, so the client must retry with the same key. For LLM and inference jobs the duplicate is a cost and consistency problem: a retried generation costs tokens again, may return different text (sampling), and may trigger tool calls twice (an agent that sends an email or refunds an order). Key the job by your own request id, store the final output, and let the worker check that record before running the model; agent tools that mutate state should take an idempotency key derived from the task step. Whether a hosted LLM provider supports an idempotency header varies, so verify in current docs rather than assuming. See also Fault Tolerance & Circuit Breakers for how retries with backoff interact with this, and Synchronous vs Asynchronous Processing for job-based APIs.
INSERT (status in_progress, scoped to the tenant and endpoint).done.done returns the stored response; a match with in_progress returns a retryable "still running" answer.A standard-library simulation with sqlite3: a broker that redelivers one message, a handler with and without an idempotency-key table, a request-hash mismatch, and two threads racing on the same key with and without a UNIQUE constraint. The barrier in case 3a only forces both threads to finish their "check" before either acts, which makes the race reproducible instead of rare.
import hashlib, json, os, sqlite3, tempfile, threading
DB = os.path.join(tempfile.mkdtemp(), "demo.db")
def conn():
c = sqlite3.connect(DB, timeout=5, isolation_level=None) # autocommit; we manage BEGIN ourselves
return c
def setup():
c = conn()
c.executescript("""
DROP TABLE IF EXISTS charges; DROP TABLE IF EXISTS idem;
CREATE TABLE charges (id INTEGER PRIMARY KEY AUTOINCREMENT, order_id TEXT, amount INTEGER);
CREATE TABLE idem (
key TEXT PRIMARY KEY, -- the UNIQUE constraint is the lock
req_hash TEXT NOT NULL,
status TEXT NOT NULL, -- 'in_progress' | 'done'
response TEXT);
""")
c.close()
def total_charged():
c = conn(); n, s = c.execute("SELECT COUNT(*), COALESCE(SUM(amount),0) FROM charges").fetchone(); c.close()
return n, s
# ---- Case 1: naive handler, broker delivers message m2 twice (at-least-once) ----
def naive_charge(c, order_id, amount):
c.execute("INSERT INTO charges(order_id, amount) VALUES (?,?)", (order_id, amount))
setup()
c = conn()
deliveries = [("m1", "order-1", 500), ("m2", "order-2", 700), ("m2", "order-2", 700), ("m3", "order-3", 300)]
for _, o, a in deliveries:
naive_charge(c, o, a)
print("1) naive handler, 4 deliveries (m2 redelivered):", total_charged(), "<- order-2 charged twice")
# ---- Case 2: idempotency-key table ----
def req_hash(payload):
return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()[:12]
def handle(c, key, payload):
h = req_hash(payload)
try:
c.execute("INSERT INTO idem(key, req_hash, status) VALUES (?,?,'in_progress')", (key, h))
except sqlite3.IntegrityError: # key already claimed
row = c.execute("SELECT req_hash, status, response FROM idem WHERE key=?", (key,)).fetchone()
if row[0] != h:
return "422 key reused with a different request body"
if row[1] == "in_progress":
return "409 original still in progress, retry later"
return "200 replayed " + row[2]
# side effect + result recorded in ONE transaction (same DB), so they commit or fail together
c.execute("BEGIN")
c.execute("INSERT INTO charges(order_id, amount) VALUES (?,?)", (payload["order"], payload["amount"]))
c.execute("UPDATE idem SET status='done', response=? WHERE key=?", ('{"charged":%d}' % payload["amount"], key))
c.execute("COMMIT")
return "200 processed"
setup()
c = conn()
for k, o, a in deliveries:
print(" ", k, handle(c, k, {"order": o, "amount": a}))
print("2) with idempotency table:", total_charged(), "<- exactly one charge per order")
print(" same key, different body:", handle(c, "m1", {"order": "order-1", "amount": 999}))
# ---- Case 3: two concurrent retries of the same request ----
def racy_check_then_act(results, i, barrier):
c = conn()
seen = c.execute("SELECT 1 FROM charges WHERE order_id='order-9'").fetchone() # check ...
barrier.wait() # both have checked
if not seen:
c.execute("INSERT INTO charges(order_id, amount) VALUES ('order-9', 1000)") # ... then act
results[i] = "charged"
else:
results[i] = "skipped"
setup()
barrier = threading.Barrier(2); results = {}
ts = [threading.Thread(target=racy_check_then_act, args=(results, i, barrier)) for i in range(2)]
[t.start() for t in ts]; [t.join() for t in ts]
print("3a) check-then-insert, no constraint:", sorted(results.values()), total_charged())
def constrained(results, i):
c = conn()
results[i] = handle(c, "req-9", {"order": "order-9", "amount": 1000})
setup(); results = {}
ts = [threading.Thread(target=constrained, args=(results, i)) for i in range(2)]
[t.start() for t in ts]; [t.join() for t in ts]
print("3b) UNIQUE key claim:", sorted(results.values()), total_charged())
# Output:
# 1) naive handler, 4 deliveries (m2 redelivered): (4, 2200) <- order-2 charged twice
# m1 200 processed
# m2 200 processed
# m2 200 replayed {"charged":700}
# m3 200 processed
# 2) with idempotency table: (3, 1500) <- exactly one charge per order
# same key, different body: 422 key reused with a different request body
# 3a) check-then-insert, no constraint: ['charged', 'charged'] (2, 2000)
# 3b) UNIQUE key claim: ['200 processed', '200 replayed {"charged":1000}'] (1, 1000)In case 3b the loser receives either the replayed response or 409 original still in progress, depending on timing; what is guaranteed is exactly one charge. Not runnable here (no output invented): the equivalent in PostgreSQL is INSERT INTO idem(key, req_hash, status) VALUES (...) ON CONFLICT (key) DO NOTHING, then check whether a row was inserted.
Q: Explain at-most-once, at-least-once and exactly-once delivery.
A: They differ by when the receiver acknowledges relative to the work. Ack before processing risks loss (at-most-once); ack after processing plus sender retries risks duplicates (at-least-once). Exactly-once delivery is not achievable over an unreliable network because the sender cannot tell "lost message" from "lost ack"; real systems combine at-least-once delivery with idempotent processing to get exactly-once effect.
Q: A vendor advertises "exactly-once". How do you respond?
A: Ask about the boundary. Kafka's transactions are atomic across Kafka topics and consumer offsets, so a consume-transform-produce loop inside Kafka can be exactly-once, but once the handler writes to an external database or API that effect is outside the transaction. So the claim holds inside the boundary and everything crossing it needs idempotent writes.
Q: Design idempotency for a POST /payments endpoint.
A: Client sends an Idempotency-Key; the server claims it with an INSERT into a table keyed by (tenant, key) storing request hash and status, performs the charge, and stores the response with the key. Retries with the same body replay the stored response, a different body is rejected, and an in-flight duplicate returns a retryable status. Keep keys for longer than the retry window (about a day is common; check your provider) and back it with a unique constraint on the order for the case where the key has expired.
Q: Why must the client generate the key rather than the server?
A: The scenario being solved is the client not knowing whether attempt one succeeded. A server-generated id is created per attempt, so attempt two gets a different id and cannot be matched to attempt one. Only a value fixed by the client before the first send survives across retries.
Q: What is the difference between natural idempotency and a dedup table?
A: Natural idempotency comes from the operation itself (set to a value, upsert on a business key, conditional update on a version), so it needs no extra state and cannot expire or race. A dedup table adds memory of processed ids to an operation that is not idempotent, at the cost of storage, a TTL decision, and the requirement that the record and the effect commit atomically.
Q: A worker consumes a message, inserts a row, then crashes before committing the offset. What happens?
A: The message is redelivered, so the handler runs again. If the insert used a unique business key with ON CONFLICT DO NOTHING, or the processed message id was recorded in the same transaction as the insert, the second run does nothing. Without either, the row is duplicated, which is the standard at-least-once failure.
Q: How do you make an LLM-backed async job safe to retry?
A: Give each job a stable id, check a results store before calling the model, and write the output (and token usage) keyed by that id when done. That prevents a second billed generation after a worker crash and returns a consistent answer instead of a differently sampled one. Any tool the agent calls that mutates state gets a key derived from the job and step.
Follow-up: Two retries of the same request arrive simultaneously. Walk me through what the database does.
A: Both try to insert the key; the unique index serializes them so one insert succeeds and the other fails with a violation (or waits on the first's lock, then fails). The winner does the work; the loser reads the row and sees in_progress (return 409 or wait) or done (replay). The point is that the constraint, not application logic, decides the winner.
Follow-up: The owner of an in-progress key crashes. How do you avoid a permanently stuck key?
A: Record a lease (started_at, or an expiry) and let a later attempt reclaim an in_progress row older than the maximum job duration. Reclaiming is only safe if the side effect is itself idempotent (downstream key, unique constraint), because the crashed attempt may have completed the effect just before dying. Choosing the lease length is a trade-off: too short risks two live owners, too long delays recovery.
Follow-up: Is HTTP PUT or DELETE always idempotent?
A: By the HTTP specification the methods are defined as idempotent, but that is a contract the implementation has to honour. A PUT handler that appends to a log, bumps a counter or sends an email on every call breaks it, and a DELETE can return 204 the first time and 404 the second, which is still the same server state but a different response. POST is not idempotent by definition, hence the key header.
Idempotency shows up wherever a design says "retry": payment and order APIs, webhook receivers (providers deliver at-least-once and typically retry on non-2xx, so dedupe on the event id), queue consumers, sagas, and outbox or CDC pipelines. A typical sizing statement: assume 1,000 payment requests per second at peak, a 24-hour key TTL, and roughly 200 bytes per row including the stored response, then the key table is about 86.4M rows and around 17 GB per day of retention (ILLUSTRATIVE; larger responses multiply this). That is comfortably a sharded SQL table with an index on the key, or Redis if the response is small and losing a key is acceptable because a business constraint still catches duplicates; the read/write ratio is roughly one insert per request plus a lookup on retries. For AI systems, an inference job queue that retries on worker failure needs the same treatment: store completed outputs per job id so a redelivered job returns the cached result instead of spending GPU time and tokens again, and pair it with the retry and circuit-breaker policies discussed elsewhere in this section.
Idempotency keys trade storage, a small latency cost on every write (one extra insert and lookup), and operational rules (TTL, scoping, cleanup) for eliminating duplicate side effects. The cost is worth it for money, inventory, notifications and anything expensive to repeat. It is the wrong tool when the operation is already naturally idempotent (do not add a table to a PUT), when the extra write would dominate a very high-throughput low-value path such as metrics ingestion (accept at-most-once or tolerate duplicates in aggregation), or when you treat it as a substitute for correctness elsewhere: a key table does not repair non-atomic side effects, does not fix an expired-TTL late retry, and does not make a non-idempotent downstream call safe. Choose the TTL and the in-progress behaviour deliberately (reject versus wait versus replay), because those are the choices that fail in production rather than the basic idea.