Fault tolerance is the set of patterns that keep your service useful while something it depends on is slow, wrong, or gone. A modern AI backend depends on a lot of things it does not control: an LLM provider's API, an embedding endpoint, a vector database, a cache, a queue, an auth service. Each has a failure rate, and the failure rates multiply through your call graph. Interviewers probe this topic because it separates people who have only written the happy path from people who have been paged at 3 a.m.: the good answers are about how a small failure grows into a big one (a slow dependency exhausting threads, retries tripling the load on a struggling server) and about the specific mechanisms that stop the growth. For a Backend or AI engineer it matters doubly, because LLM calls are slow, expensive, rate-limited, and frequently overloaded, which makes every one of these failure modes more likely than with a typical database call.
Fault tolerance is designing a system so that when one part of it breaks, slows down, or misbehaves, the rest of the system keeps working instead of collapsing too. No system depends on perfectly reliable pieces — servers crash, networks hiccup, other companies' services go down — so a fault-tolerant system assumes failure will happen and plans for what to do when it does, instead of hoping it never will.
A circuit breaker is a safety switch that software uses to stop calling a service that has started failing, instead of continuing to hammer it. Just like the circuit breaker in a household electrical panel trips to cut power before wires overheat, a software circuit breaker "trips open" after enough failures, so callers get an instant, safe "not available right now" instead of a long, wasted wait, which also gives the struggling service room to recover.
Picture a single overworked cashier at a busy store. If every customer keeps walking up and re-asking the same question the moment the cashier doesn't answer instantly, the line backs up and nobody gets served. A good store manager notices the cashier is overwhelmed and temporarily redirects new customers to self-checkout, instead of letting everyone pile up. A circuit breaker is that manager: it notices something is struggling and reroutes traffic away until things recover.
Service handles 200 requests/second, has 500 worker threads / connections.
In-flight requests = arrival rate x time each request holds a worker (Little's law).
Dependency state Latency seen by caller In flight (200/s) Pool of 500
--------------------------- ------------------------- ------------------ ------------
healthy 50 ms 10 fine
degraded 2 s 400 nearly full
hung, 30 s client timeout 30 s 6,000 EXHAUSTED
down, connection refused 5 ms (fails fast) 1 fine
Once the pool is exhausted, requests that never touch the sick dependency
(health endpoint, cached answers, other features) queue behind it and fail too.A dependency that is cleanly down is the easy case: the connection is refused in milliseconds, the caller gets an error, and it can fall back. A dependency that is slow is the dangerous case because it fails the caller's resources instead of the caller's request. Every request holds a thread, a socket, memory and often a database connection for the whole wait, so the number of concurrent requests rises with latency even though traffic is flat. That is the arithmetic above (an ILLUSTRATIVE calculation, not a measurement), and it is why "add a timeout to every outbound call" is the first rule of resilience.
Timeouts need thought, not a default. Separate the connect timeout (short, a second or less, since a healthy peer accepts quickly) from the read/request timeout (sized to a high percentile of healthy latency plus headroom, not to "whatever finishes"), and above both keep an overall deadline for the whole user request that is propagated downstream so a deep call does not keep working after the caller gave up. LLM calls add a twist: a streamed completion can legitimately run for many seconds, so a single total timeout is a blunt tool. Use a time-to-first-token timeout (if nothing starts in a few seconds, the provider is stuck) plus an idle timeout between chunks, and an overall cap tied to the maximum output length you allow. See Timeouts, Retries & Backoff for the network-level mechanics and Streaming Responses for how streams change what "done" means.
Retry only what is worth retrying:
yes: timeouts, connection resets, 502/503/504, 429 (honour Retry-After), provider "overloaded" errors
no: 400/422 (your request is wrong), 401/403 (credentials), content-policy refusals
careful: non-idempotent writes -> only with an idempotency key
Backoff schedule, full jitter: delay = uniform(0, min(cap, base * 2^attempt))
attempt 0: ceiling 0.5 s attempt 1: 1 s attempt 2: 2 s attempt 3: 4 s attempt 4+: 8 s (cap)
Why jitter? 1,000 clients time out together -> without jitter they retry together at
+1 s, +2 s, +4 s, each wave a synchronized spike. Jitter spreads each wave over the window.
Retry amplification through layers (each layer makes 3 attempts):
client -> API gateway -> orchestrator -> LLM provider
3 x 3 x 3 = 27 provider calls for ONE user click when the provider is failing.Retries are the right tool for transient faults: a dropped connection, a single overloaded replica, a brief 503. For those, one or two retries cheaply convert a visible error into a slightly slower success. They are the wrong tool for a persistent fault, where every retry is more load on a system that is already failing. That is a retry storm: the dependency slows down, callers time out and retry, the retries raise the load, which slows it further, which causes more timeouts. It can even keep a service down after the original cause is gone (a metastable failure), because the retry traffic alone is enough to keep it saturated.
Exponential backoff slows the retry rate as failures continue, and jitter is what makes it work across many clients; the widely cited "full jitter" variant draws the delay uniformly from zero up to the exponential ceiling. A retry budget adds a global brake: allow retries only up to some fraction of recent normal traffic (the exact default varies by library; figures from roughly 10 to 20 percent are common), so that even a total outage multiplies load by roughly 1.1x instead of 4x. Retry at one layer only, usually the edge closest to the user or the layer that has the most context, and pass a "do not retry" signal downstream. If a retried operation has side effects, it must be idempotent or carry an idempotency key; see Idempotency & Exactly-Once Semantics. When the server tells you to wait (a 429 with Retry-After), obey it instead of guessing; Rate Limiting & Throttling covers the server side of that conversation.
Circuit breaker state machine (one breaker per dependency, ideally per endpoint):
failure rate over window >= threshold
+--------+ ------------------------------------> +--------+
| CLOSED | | OPEN | calls fail instantly
| (pass) | <--+ | (fail | (fallback runs)
+--------+ | | fast) |
^ | probe succeeds +--------+
| | | cooldown elapsed
| +-----------+ v
+---| HALF-OPEN | <-------------------------------+
| 1..N test | probe fails -> back to OPEN (new cooldown)
+-----------+
Bulkhead: separate pools so one slow dependency cannot take everything
[ LLM calls: 40 slots ] [ vector DB: 60 slots ] [ everything else: 400 slots ]
Load shedding: when queue depth or latency crosses a limit, reject cheap and early
(HTTP 503 / 429) instead of accepting work you cannot finish before its deadline.A circuit breaker watches the outcomes of calls to one dependency. While closed it passes everything through. When the failure rate in a rolling window crosses a threshold (and a minimum number of calls has been seen, so two failures out of three do not trip it), it opens: calls fail immediately without touching the dependency, which protects your threads and gives the dependency room to recover. After a cooldown it goes half-open and lets a small number of probe calls through; success closes it, failure re-opens it. The knobs matter: a window that is too small trips on noise, a cooldown that is too short keeps hammering a sick service, and a breaker shared across unrelated endpoints will open for all of them because one is failing. Count timeouts and slow calls (over some latency threshold) as failures, not just errors, or the breaker will stay closed through the most dangerous failure mode above. See Circuit Breaker for the networking view.
Bulkheads apply the ship-compartment idea: give each dependency its own bounded concurrency pool (semaphores, separate thread pools, separate connection pools, or separate replicas of the service for different tenants), so a hung LLM provider can consume its 40 slots but not the whole service. Bulkheads and breakers are complements: the bulkhead limits the damage while you are still discovering the dependency is sick, the breaker stops you from spending the slots at all once you know. Load shedding is the same thinking applied to your own capacity: when you are overloaded, reject some work quickly and cheaply, prioritising by importance (interactive chat over batch re-indexing, paying tiers over free tiers), because a queue that grows past what can be served within the deadline turns all requests into timeouts. It is different from rate limiting, which enforces fairness per client regardless of current health.
Degradation ladder for a RAG chat endpoint (try top to bottom, stop at the first that works):
1. Primary model, full retrieval (normal)
2. Same model via a second provider / region (provider failover)
3. Smaller / cheaper model, fewer retrieved chunks (lower quality, lower latency)
4. Semantic-cache hit for a near-duplicate question (fast, possibly stale)
5. Retrieval only: show top passages + links, no generation
6. Canned answer / status message / "queued, we will email you"
Vector DB down -> keyword search fallback (BM25), or serve cached retrieval results
LLM slow, not down -> hedge to a second provider after the p95 latency (idempotent calls only)Circuit breakers only help if opening one leads somewhere useful. A fallback is the alternate answer served when the primary path fails, and designing it is a product decision as much as an engineering one. For an LLM feature, a ladder like the one above lets quality degrade in steps instead of falling off a cliff: switch provider or region, then drop to a smaller model, then serve from a cache, then show retrieved passages without generated prose, and finally a clear message. Each rung has to be tested, because a fallback that has never run in production usually does not work when first needed, and each rung should be visible in metrics and in the response (a flag or a UI note) so you and the user know the answer is degraded. Some paths have no honest fallback: an unverified generated answer to a medical or financial question is worse than an error, so in those flows the right degradation is to refuse. Related design work: LLM Gateway & Multi-Provider Routing and Semantic Caching for LLMs; for the retrieval side see Vector Databases.
Hedged requests attack tail latency rather than failure: send the request, and if there is no response by roughly the p95 latency, send a second copy to another replica or provider and use whichever answers first, cancelling the other. Since only the slowest few percent exceed p95, the extra load is small (on the order of 5 percent by construction), yet the tail improves noticeably. Two conditions apply: the operation must be safe to run twice (reads, embeddings, idempotent writes), and the extra cost must be acceptable, which for long, paid LLM generations often it is not, so hedging is usually reserved for embeddings, retrieval, and short calls.
Probe Question it answers Action on failure Should it call dependencies?
--------------- ------------------------------------- -------------------------- ----------------------------
liveness Is the process wedged / deadlocked? restart the container No
readiness Can this instance take traffic now? remove from load balancer Only local state (warm-up done,
not overloaded); be careful
deep / synthetic Does the whole path work end to end? alert a human, dashboard Yes, but run it OUT of the
(real query through the vector DB (rarely auto-remove) load-balancer path, at low rate
and a tiny LLM call)
Anti-pattern: readiness = "can I reach the vector DB?" -> vector DB blips -> EVERY instance
reports not-ready at once -> load balancer has zero healthy backends -> total outage
that is worse than serving degraded answers.Health checks are themselves a failure amplifier when designed badly. In Kubernetes terms (probe names are Kubernetes-specific, but the ideas are general) a liveness probe should only detect a stuck process, since its failure action is a restart; a readiness probe gates traffic and should reflect whether this instance can serve, not whether a shared dependency is healthy, because a shared failure would drain the entire fleet at once. A deep check that exercises the real path is valuable for alerting and for canaries, but should not drive automatic removal from rotation. A better pattern is to let the circuit breaker's state be the signal: an open breaker means "this dependency is bad", and the fallback path keeps the instance serving.
You do not know your resilience is real until you break things on purpose. Chaos testing injects the faults the design claims to survive: kill instances, add latency, return errors or malformed responses, drop packets, exhaust a rate limit, black-hole the vector database. Tools such as Chaos Monkey-style instance killers, Chaos Mesh or Litmus on Kubernetes, and Toxiproxy for network faults exist for this, but a simple fault-injecting wrapper around your own client works too. Start in staging or with a tiny blast radius, state the hypothesis first ("with the provider returning 503, p99 stays under 3 s and the fallback rate rises"), and treat a surprise as a finding. For LLM systems, include provider-specific faults: 429s with and without Retry-After, streams that stall mid-response, truncated or malformed JSON output, and very slow first tokens.
Retry-After), and tries again, always within the remaining request deadline and using an idempotency key if the call has side effects.# Seeded discrete-event simulation: flaky dependency vs three client strategies.
import heapq, random
from collections import deque
RPS, DURATION = 80, 60 # 80 new requests/second for 60 simulated seconds
CAPACITY = 100 # dependency serves at most 100 attempts per second
OUTAGE = (10.0, 25.0) # dependency fails every call in [10s, 25s)
DEADLINE = 6.0 # client gives up (or falls back) 6s after arrival
MAX_ATTEMPTS = 4
class Dependency:
def __init__(self, scenario, rnd):
self.per_sec, self.attempts = {}, 0
self.scenario, self.rnd = scenario, rnd
def call(self, t):
self.attempts += 1
s = int(t)
self.per_sec[s] = self.per_sec.get(s, 0) + 1
if self.scenario == "blip":
return self.rnd.random() >= 0.20 # 20% of calls fail at random
if OUTAGE[0] <= t < OUTAGE[1]:
return False # hard outage
return self.per_sec[s] <= CAPACITY # attempts beyond capacity are rejected
class Breaker:
def __init__(self, window=50, min_calls=20, rate=0.5, cooldown=5.0):
self.results, self.window, self.min_calls = deque(maxlen=window), window, min_calls
self.rate, self.cooldown = rate, cooldown
self.state, self.opened_at, self.probe_out = "closed", 0.0, False
self.trips = 0
def allow(self, t):
if self.state == "open" and t - self.opened_at >= self.cooldown:
self.state, self.probe_out = "half_open", False
if self.state == "closed": return True
if self.state == "half_open" and not self.probe_out:
self.probe_out = True; return True # exactly one probe at a time
return False
def record(self, t, ok):
if self.state == "half_open":
if ok: self.state = "closed"; self.results.clear()
else: self.state, self.opened_at = "open", t
return
self.results.append(ok)
if len(self.results) >= self.min_calls and self.results.count(False) / len(self.results) >= self.rate:
self.state, self.opened_at = "open", t; self.trips += 1
def run(strategy, scenario, seed=42):
rnd = random.Random(seed)
dep, br = Dependency(scenario, random.Random(seed + 1)), Breaker()
events, out = [], dict(ok=0, fallback=0, failed=0)
for i in range(RPS * DURATION):
heapq.heappush(events, (i / RPS + rnd.random() / RPS, i, 1, i / RPS))
budget_tokens, sent, retries = 0.0, 0, 0
while events:
t, rid, attempt, t0 = heapq.heappop(events)
if t - t0 > DEADLINE: # deadline passed while waiting
out["fallback" if strategy == "breaker" else "failed"] += 1; continue
if strategy == "breaker" and not br.allow(t):
out["fallback"] += 1; continue # fail fast, canned answer
if attempt == 1: sent += 1; budget_tokens = min(budget_tokens + 0.1, 20)
ok = dep.call(t)
if strategy == "breaker": br.record(t, ok)
if ok: out["ok"] += 1; continue
if attempt >= MAX_ATTEMPTS:
out["fallback" if strategy == "breaker" else "failed"] += 1; continue
if strategy == "naive":
delay = 0.0 # retry immediately
else:
delay = rnd.uniform(0, min(4.0, 0.25 * 2 ** (attempt - 1))) # full jitter
if strategy == "budget":
if budget_tokens < 1: out["failed"] += 1; continue # budget spent: no retry
budget_tokens -= 1
retries += 1
heapq.heappush(events, (t + delay, rid, attempt + 1, t0))
peak = max(dep.per_sec.values())
total = RPS * DURATION
print(f"{strategy:8s} attempts={dep.attempts:6d} amp={dep.attempts/total:4.2f}x peak/s={peak:4d} "
f"ok={out['ok']/total:6.1%} fallback={out['fallback']/total:6.1%} failed={out['failed']/total:6.1%}"
+ (f" trips={br.trips}" if strategy == "breaker" else ""))
for scenario in ("blip", "outage"):
print(f"--- scenario: {scenario} ({RPS*DURATION} requests, capacity {CAPACITY}/s) ---")
for s in ("naive", "jitter", "budget", "breaker"):
run(s, scenario)
# Real output from python3:
# --- scenario: blip (4800 requests, capacity 100/s) ---
# naive attempts= 5933 amp=1.24x peak/s= 108 ok= 99.9% fallback= 0.0% failed= 0.1%
# jitter attempts= 5936 amp=1.24x peak/s= 111 ok= 99.9% fallback= 0.0% failed= 0.1%
# budget attempts= 5279 amp=1.10x peak/s= 90 ok= 88.9% fallback= 0.0% failed= 11.1%
# breaker attempts= 5936 amp=1.24x peak/s= 111 ok= 99.9% fallback= 0.1% failed= 0.0% trips=0
# --- scenario: outage (4800 requests, capacity 100/s) ---
# naive attempts= 8400 amp=1.75x peak/s= 320 ok= 75.0% fallback= 0.0% failed= 25.0%
# jitter attempts= 8482 amp=1.77x peak/s= 334 ok= 76.3% fallback= 0.0% failed= 23.7%
# budget attempts= 4939 amp=1.03x peak/s= 106 ok= 75.0% fallback= 0.0% failed= 25.0%
# breaker attempts= 3607 amp=0.75x peak/s= 80 ok= 74.6% fallback= 25.4% failed= 0.0% trips=1
# Reading the results (this simulation is deliberately simple, so treat the shapes, not the decimals, as the lesson):
# - blip (20% random failures): naive and jittered retries both reach ~99.9% success at 1.24x load. Retries WORK for transient faults.
# The 10% retry budget is too tight for a 20% failure rate, so it gives up on 11.1% of requests; a budget is a deliberate cap, not free.
# - outage (10 s hard down): no retry policy can help while the dependency is down. Naive and jitter multiply load to 320-334 attempts/s
# against a dependency that can serve 100. The budget holds load at ~1.03x, and the breaker cuts it BELOW normal (0.75x) by failing
# fast and serving a canned fallback for the 25% of requests that arrived during the outage.# Slow is worse than down: in-flight requests = arrival rate x latency (Little's law)
rate = 200 # requests/second into the service
pool = 500 # worker threads / connections available
for label, latency_s in [("healthy", 0.05), ("degraded", 2.0), ("hung, 30s timeout", 30.0)]:
inflight = rate * latency_s
print(f"{label:18s} latency={latency_s:5.2f}s -> in flight={inflight:7.0f} pool={pool} "
f"{'EXHAUSTED' if inflight > pool else 'ok'}")
print("fail-fast (down, refused in 5ms):", rate * 0.005, "in flight")
# Exponential backoff with full jitter: delay = uniform(0, min(cap, base * 2**attempt))
import random
random.seed(1)
base, cap = 0.5, 8.0
for attempt in range(5):
ceiling = min(cap, base * 2 ** attempt)
draws = [round(random.uniform(0, ceiling), 2) for _ in range(3)]
print(f"attempt {attempt}: ceiling={ceiling:4.1f}s sample delays={draws}")
# Real output from python3:
# healthy latency= 0.05s -> in flight= 10 pool=500 ok
# degraded latency= 2.00s -> in flight= 400 pool=500 ok
# hung, 30s timeout latency=30.00s -> in flight= 6000 pool=500 EXHAUSTED
# fail-fast (down, refused in 5ms): 1.0 in flight
# attempt 0: ceiling= 0.5s sample delays=[0.07, 0.42, 0.38]
# attempt 1: ceiling= 1.0s sample delays=[0.26, 0.5, 0.45]
# attempt 2: ceiling= 2.0s sample delays=[1.3, 1.58, 0.19]
# attempt 3: ceiling= 4.0s sample delays=[0.11, 3.34, 1.73]
# attempt 4: ceiling= 8.0s sample delays=[6.1, 0.02, 3.56]Q: Why is a slow dependency often worse than a dead one?
A: A dead dependency fails fast, so the caller frees its resources and can fall back. A slow one holds a thread, socket and memory per request for the whole wait, and by Little's law the number in flight equals arrival rate times latency, so at 200 requests/s a 30 s hang means about 6,000 stuck requests against a pool of maybe 500. Once the pool is full, unrelated features fail too. The fixes are tight timeouts, bulkheads, and treating slow calls as failures in the breaker.
Q: How do you retry safely?
A: Retry only errors that are plausibly transient (timeouts, connection resets, 502/503/504, 429 honouring Retry-After), not 4xx caused by a bad request. Use exponential backoff with jitter, a low attempt cap, a retry budget shared across the client, and an overall deadline. Retry at one layer of the stack, and only retry operations that are idempotent or carry an idempotency key, otherwise a retry after a lost response can duplicate a charge or a side effect.
Q: What is jitter for, and what is full jitter?
A: Without randomness, clients that failed together retry together, so each backoff step is a synchronized spike (the thundering herd). Jitter randomizes the delay so the retry load is spread over the window. Full jitter draws the delay uniformly between zero and the exponential ceiling, which spreads best at the price of some individual retries happening quickly; other variants keep a minimum delay to avoid that.
Q: Explain the states of a circuit breaker and what could go wrong with the configuration.
A: Closed passes calls and tracks results; open fails calls immediately for a cooldown; half-open lets a few probes through to decide whether to close or re-open. Too small a window or too low a threshold trips on noise (in a variant of my simulation with a 20-call window and 10-call minimum, 20% random failures tripped it 3 times in a 60 s run, while the 50-call window with a 20-call minimum never did); too short a cooldown keeps pounding a recovering service; one breaker for many endpoints lets one bad endpoint disable the rest.
Q: What is the difference between a circuit breaker, a bulkhead, and rate limiting?
A: A breaker reacts to the observed health of a dependency and stops calling it. A bulkhead bounds how much of your own resources (threads, connections, concurrency) a dependency or tenant can use, whether or not it is failing. Rate limiting caps how much a client is allowed to send regardless of health, for fairness and cost. They are layered: bulkhead limits damage, breaker stops the calls, rate limiting and load shedding protect capacity.
Q: Your LLM provider starts returning 429s and 503s. What does the system do?
A: Honour Retry-After and back off with jitter within a request deadline, open the breaker for that provider when the failure rate crosses the threshold, and route to the next rung: a second provider or region, then a smaller model, then a semantic cache hit, then retrieval-only results, then a clear "try again" message. Shed low-priority traffic (batch jobs, free tier) first so interactive users keep capacity, and alert on the fallback rate. Provider error semantics differ, so check the provider's current docs for which codes are retryable.
Q: The vector database is down. Does the chatbot go down?
A: It should not, but the fallback is a quality decision. Options are a keyword index (BM25) over the same corpus, cached retrieval results for popular questions, or answering only from cache. If none of them can ground the answer, the honest behaviour is to say the knowledge base is unavailable rather than let the model answer from memory and present it as grounded. Retrieval failing is a reason to degrade, not to hallucinate.
Follow-up: A team adds retries at the client, the gateway, and the service. What happens in an outage?
A: Attempts multiply: with 3 attempts at each of three layers, a failing dependency sees up to 27 calls per user request, and timeouts stack so users wait even longer. Fix it by choosing one retrying layer, adding a retry budget, propagating a deadline and a "do not retry" header downstream, and having lower layers fail fast.
Follow-up: Why not make readiness fail when the database is unreachable?
A: Because a shared dependency failure would then mark every instance not-ready at the same moment, the load balancer would have no backends, and you would turn a degraded service into a total outage (and restarts, if liveness were tied to it, would add cold-start load). It is generally better to stay ready, serve what you can through fallbacks, and expose dependency health through breaker state, metrics and alerts. Readiness should reflect this instance's ability to serve.
Follow-up: How would you prove your fallbacks work?
A: Run them: chaos tests that inject provider 503s, added latency, stalled streams, and a blacked-out vector DB in staging (and, carefully, in production with a small blast radius), with a stated hypothesis such as "p99 under 3 s, fallback rate under 40 percent, no errors on unrelated endpoints". Also exercise the rungs regularly in normal traffic, for example by sending a small percentage of requests down the fallback path, because an untested fallback is generally an unreliable one.
In an AI backend the resilience layer usually lives in the LLM gateway and in the retrieval client, and interviewers expect you to name the pieces rather than say "we will add retries". A typical answer: per-provider timeouts (time-to-first-token plus idle plus overall), a breaker and a bulkhead per provider and per vector-DB cluster, a retry budget with jittered backoff at the gateway only, a fallback ladder (secondary provider, smaller model, semantic cache, retrieval-only, canned message), priority-based load shedding, and health checks that keep readiness shallow. Back it with a rough estimate stated as an assumption: at, say, 500 requests/s and a naive 4 attempts, a provider outage turns into up to 2,000 provider calls/s, which is exactly the load that keeps a rate-limited or recovering provider down, while a 10 percent retry budget caps it near 550 (ILLUSTRATIVE). Add the SLO framing from Capacity Planning & SLOs (an error budget tells you how much degradation you can afford), and close with observability: breaker state, retry rate, fallback rate, shed count, and per-provider latency percentiles on the dashboard.
Every mechanism here buys resilience by spending something. Retries spend load and latency to fix transient faults, and cause outages when misapplied to persistent ones. Breakers spend correctness at the margin (a healthy request can be rejected while the breaker is open, and thresholds can trip on noise) and add per-dependency state and tuning. Bulkheads spend utilisation: reserved slots sit idle while other pools are starved. Fallbacks spend quality and cost engineering effort, and they add code paths that are exercised rarely and therefore break silently. Hedging spends extra requests for lower tail latency. The mechanisms are the wrong choice when the operation is not safe to repeat and has no idempotency key, when the dependency is a single point of failure with no alternative (then fail clearly and fast rather than pretending), when a simple deadline and a clear error message is all the product needs, or when the added complexity exceeds the risk for a low-traffic internal tool. Start with timeouts, one disciplined retry policy, and load shedding; add breakers, bulkheads, and multi-provider fallbacks as the traffic and the cost of an outage justify them, and measure the fallback rate so you know they are working.