A real LLM feature is rarely one prompt in and one string out. It is a small program: retrieve some documents, build a prompt, call a model, parse the answer, maybe call a tool and loop, then log what happened. Orchestration frameworks such as LangChain, LlamaIndex, LangGraph, Semantic Kernel, and DSPy package the recurring parts of that program. Interviewers probe this topic to see whether you can reason about an abstraction layer instead of cargo-culting it: what does it actually buy, what does it hide, how do you debug it at 2 a.m., and when would you delete it. For a Backend or AI engineer, the useful skill is to see every framework as a way to structure a directed graph of steps with state, retries, and tracing, whether you adopt a library or write forty lines yourself.
An LLM orchestration framework is a toolkit that helps developers wire together the multiple steps a real AI feature usually needs, rather than just one prompt and one reply. Behind almost any useful AI assistant there's a small sequence of actions: look something up, build a well-formed question for the model, send it, read the answer, maybe use a tool like a calculator or a search engine, and log what happened for debugging. Orchestration frameworks provide ready-made pieces for that sequence — connecting to different AI models, managing conversation memory, and handling errors — so a team doesn't have to build all of that plumbing from scratch for every new feature.
Picture a restaurant kitchen during a dinner rush. A waiter doesn't cook the whole meal alone — the order goes to the right station (grill, salad, dessert), each cook does their one job, and a expediter checks the plate before it goes out, sending anything wrong back to be redone. An orchestration framework plays the role of that kitchen's workflow: it makes sure each step of preparing an answer happens in the right order, by the right "station," with a system for catching and fixing mistakes before the final plate reaches the customer.
+------------------- orchestration layer -------------------+
request -> | step: retrieve -> step: prompt -> step: model -> parse |
| | | | | |
| vector DB template/registry provider API schema |
| | |
| tool call? -- yes --> run tool --+ |
| | | |
| +---- no ----> final answer <-- loop |
| |
| cross-cutting: state/memory | retries | timeouts | fallback |
| tracing | streaming | cost + token accounting |
+---------------------------------------------------------------+
Building blocks common to most frameworks (names differ):
runnable/step one unit: input -> output
chain / graph how steps connect (linear, branching, looping)
tool calling model requests a function; the framework executes and feeds the result back
memory / state conversation history or a typed state object carried between steps
retriever embed query -> search index -> return chunks
callbacks/tracing hooks fired before and after each stepStrip away the vocabulary and the layer solves four problems. It gives steps a common interface so they can be composed and swapped (a different model or vector store behind the same shape). It runs the control flow, including the agent loop where the model asks for a tool, the runtime executes it, and the result goes back in. It carries state between steps, whether that is chat history or a typed object. And it provides hooks so each step can be traced. None of this is conceptually hard, which is exactly why the build-versus-adopt question is real: the value is in the integrations, the shared vocabulary, and the ecosystem of ready-made connectors, not in some capability you could not write yourself.
Framework Center of gravity (generally) Typical reason to pick it
--------------- ---------------------------------------------------- ---------------------------------
LangChain general-purpose components + integrations (models, breadth of connectors, fast
retrievers, tools, output parsers), composable steps prototyping, large community
LangGraph agents and workflows as an explicit graph of nodes stateful, looping, branching flows
over shared state, with persistence/checkpointing that need control and resumability
LlamaIndex data-centric: ingestion, indexing, query engines, RAG-heavy apps where the
retrieval over your own documents indexing/retrieval side dominates
Semantic Kernel Microsoft's SDK: plugins/functions the model can call, .NET / Java / Python shops,
planning and orchestration Azure-adjacent environments
DSPy programs written as modules with declared signatures; when you want to optimize
an optimizer tunes prompts/examples against a metric prompts against a metric, not
hand-write them
Also seen: Haystack, provider agent SDKs, and plain code with a provider client.Treat the table as a working map, not a spec. These libraries evolve quickly, split and merge packages, and rename concepts, so specific class names, features, and recommended patterns change between releases; verify against current documentation before relying on any detail. The stable distinction is philosophy. Component-and-chain libraries emphasize composition and integrations, graph runtimes emphasize explicit control flow and state, data-centric libraries emphasize retrieval quality over your corpus, enterprise SDKs emphasize plugins and language support, and DSPy-style tools shift effort from writing prompts to defining metrics and letting an optimizer search. Choosing among them is mostly choosing which of those centers matches the dominant complexity of your application.
Linear chain (fine for fixed pipelines):
retrieve -> prompt -> generate -> parse
Graph / state machine (needed once there are loops or branches):
+--------------------+
v |
START -> plan -> call_model -> needs_tool? --yes--> run_tool ---+
| |
no steps >= MAX_STEPS? --> abort/escalate
v
validate -> ok? --yes--> END
|
no --> repair (bounded) --> call_model
State (typed, explicit): {messages, retrieved, tool_results, step_count, error}
Every edge is a function of state; every node writes to state; each transition can be logged
and (with persistence) checkpointed so a crashed run can resume or a human can approve a step.An agent is a loop, and loops need guard rails that a linear chain never did: a maximum step count so a confused model cannot spin forever (and burn money), an explicit terminal condition, bounded repair attempts, and an escalation path. Modelling the flow as a state machine or graph makes those rules visible: nodes are operations, edges are conditions over a shared state object, and the whole run is a sequence of state transitions you can log, replay, and unit test node by node. That is the main reason graph-style runtimes exist. Persisting state between transitions (checkpointing) also enables human approval steps and resuming after a crash, at the cost of designing state that serializes cleanly. For tool semantics and safety see agentic systems and tool use.
Per-step policy (declared on the step, not scattered through code):
timeout per attempt, so one slow provider call cannot eat the whole request
retries only for transient errors (timeouts, 429, 5xx), with exponential backoff + jitter
fallback cheaper/other model, cached answer, or a canned "could not answer"
idempotency a retried step that has side effects (send email, charge) must be safe to repeat
Worst-case latency = attempts x timeout + sum(backoffs) + fallback time
Retries multiply load on a struggling dependency: pair with circuit breakers and a total deadline.
Trace record per step (one span each):
run_id, step, attempt, status, latency_ms, model, prompt_version,
input_tokens, output_tokens, error, parent_spanRetry policy is where orchestration most often goes wrong at scale. Retrying every error hides bugs, retrying without backoff and jitter turns a brief provider blip into a self-inflicted outage, and stacking retries at several layers (client, framework, gateway) multiplies attempts silently, so a per-request deadline and one owner of the retry policy matter. Only transient failures are worth retrying; a schema-validation failure deserves a bounded repair attempt, not blind repetition; and side-effecting steps need idempotency keys. See fault tolerance and circuit breakers. For observability, one trace per request with one span per step (model, prompt version, tokens, latency, error) is what makes a multi-step failure debuggable; frameworks generally expose callback or hook mechanisms, and standards such as OpenTelemetry can carry the data, but check what a given framework emits by default. More on this in observability for AI systems.
hand-rolled (client SDK + your code) framework
first demo slower to first result fast (ready-made connectors)
what is sent you see the exact prompt/request may be assembled behind layers
debugging stack trace = your code stack trace crosses library internals
upgrades you control the pace library changes may force rewrites
provider swap your own thin interface often easier IF the abstraction holds
team onboarding read your code shared vocabulary, docs, examples
lock-in low moderate: prompts, state, tracing formats
Decision aids:
1 model call + parse -> plain code
fixed 3-5 step RAG pipeline -> plain code or light library use
loops, branching, checkpoints, HITL -> graph runtime, or your own state machine
many integrations (loaders, stores) -> adopt for the connectors
prompt quality is the bottleneck -> consider metric-driven optimization toolsThe cost of an abstraction is paid at debugging time. When an answer is wrong you need the exact final prompt, the raw model response, and which step produced what; if a framework assembles the prompt through several layers you must be able to see it, so insist on tracing before adopting. The other cost is churn: fast-moving APIs mean upgrades can be breaking, and pipelines woven through framework-specific classes are expensive to migrate. A common mitigation is a thin boundary of your own (an interface for the model, the retriever, and the tools) so the framework sits behind it and can be replaced. Adopt when connectors and shared structure save real time; write your own when the flow is small and the failure modes matter more than the feature list. Neither is free.
This stdlib-only program is a minimal graph executor: steps, a conditional edge, bounded retries, a fallback, and a trace row per attempt. The primary "model" is a fake that fails its first two calls, so the fallback path is exercised deterministically. It is a teaching sketch, not how any particular framework is implemented.
# Python 3, standard library only: a minimal pipeline/graph executor with retries,
# fallback and per-step tracing. The "model" is a fake deterministic function.
class TransientError(Exception): pass
class FakeModel:
"""Deterministic: primary fails on its first 2 calls, fallback always works."""
def __init__(self, name, fail_first=0):
self.name, self.fail_first, self.calls = name, fail_first, 0
def __call__(self, prompt):
self.calls += 1
if self.calls <= self.fail_first:
raise TransientError(f"{self.name} 503 (call {self.calls})")
return f"[{self.name}] answer to: {prompt[:30]}"
class Trace:
def __init__(self): self.rows = []
def add(self, step, attempt, status, detail=""):
self.rows.append((step, attempt, status, detail))
def show(self):
for i, (s, a, st, d) in enumerate(self.rows):
print(f"{i:02d} step={s:<9} attempt={a} status={st:<9} {d}")
class Step:
def __init__(self, name, fn, retries=0, fallback=None):
self.name, self.fn, self.retries, self.fallback = name, fn, retries, fallback
def run_step(step, state, trace):
for attempt in range(1, step.retries + 2):
try:
out = step.fn(state)
trace.add(step.name, attempt, "ok")
return out
except TransientError as e:
trace.add(step.name, attempt, "error", str(e))
# a real system would sleep with exponential backoff + jitter here
if step.fallback:
out = step.fallback(state)
trace.add(step.name, "-", "fallback", "used fallback")
return out
raise RuntimeError(f"step {step.name} failed with no fallback")
def run_graph(steps, edges, state, trace):
"""Follow edges: each edge is a function state -> next step name (or None to stop)."""
name = steps[0].name
by_name = {s.name: s for s in steps}
while name:
state[name] = run_step(by_name[name], state, trace)
name = edges[name](state)
return state
primary, backup = FakeModel("primary", fail_first=2), FakeModel("backup")
docs = {"refund": "Refunds are allowed within 14 days.", "shipping": "Shipping takes 3-5 days."}
def retrieve(st):
q = st["question"].lower()
return [t for k, t in docs.items() if k in q] or []
def generate(st):
ctx = " ".join(st["retrieve"]) or "(no context)"
return primary(f"{ctx} Q: {st['question']}")
def generate_fallback(st):
return backup(f"{' '.join(st['retrieve'])} Q: {st['question']}")
def refuse(st):
return "I could not find that in the knowledge base."
steps = [Step("retrieve", retrieve),
Step("generate", generate, retries=1, fallback=generate_fallback),
Step("refuse", refuse)]
edges = {"retrieve": lambda st: "generate" if st["retrieve"] else "refuse", # conditional edge
"generate": lambda st: None,
"refuse": lambda st: None}
for q in ("What is the refund window?", "Do you sell gift cards?"):
trace = Trace()
result = run_graph(steps, edges, {"question": q}, trace)
final = result.get("generate") or result.get("refuse")
print("Q:", q); trace.show(); print("=>", final); print()
print("primary calls:", primary.calls, "| backup calls:", backup.calls)
# Real output from python3:
# Q: What is the refund window?
# 00 step=retrieve attempt=1 status=ok
# 01 step=generate attempt=1 status=error primary 503 (call 1)
# 02 step=generate attempt=2 status=error primary 503 (call 2)
# 03 step=generate attempt=- status=fallback used fallback
# => [backup] answer to: Refunds are allowed within 14
#
# Q: Do you sell gift cards?
# 00 step=retrieve attempt=1 status=ok
# 01 step=refuse attempt=1 status=ok
# => I could not find that in the knowledge base.
#
# primary calls: 2 | backup calls: 1The second program does the worst-case latency arithmetic for a retried step. The timeout, backoff, and deadline values are assumptions chosen for illustration, not measurements of any provider.
# Python 3: worst-case latency of a step with per-attempt timeout, retries and backoff.
# Numbers are ILLUSTRATIVE assumptions, not measurements.
def worst_case(timeout_s, retries, base_backoff_s, then_fallback_timeout_s=0.0):
attempts = retries + 1
backoffs = sum(base_backoff_s * 2**i for i in range(retries)) # 1x, 2x, 4x ...
return attempts * timeout_s + backoffs + then_fallback_timeout_s
for retries in (0, 2, 4):
w = worst_case(timeout_s=10, retries=retries, base_backoff_s=0.5, then_fallback_timeout_s=10)
print(f"retries={retries}: worst case before fallback answers = {w:.1f} s, fits a 30 s deadline: {w <= 30}")
# Real output from python3:
# retries=0: worst case before fallback answers = 20.0 s, fits a 30 s deadline: True
# retries=2: worst case before fallback answers = 41.5 s, fits a 30 s deadline: False
# retries=4: worst case before fallback answers = 67.5 s, fits a 30 s deadline: FalseQ: What problem do LLM orchestration frameworks solve?
A: They standardize the plumbing around model calls: a common step interface, composition of steps into chains or graphs, tool-calling loops, memory or state, retrieval integrations, and tracing hooks. That speeds up prototypes and gives a team shared vocabulary and ready-made connectors. What they do not add is capability you could not hand-write, so the honest answer is that they trade control and transparency for development speed.
Q: When would you not use a framework?
A: When the task is a single model call plus parsing, or a small fixed pipeline, because the framework adds dependency churn and indirection without saving much code. Also when the hot path needs tight control over the exact prompt, latency, and error handling. A thin wrapper around the provider SDK, with your own retry, timeout, and tracing, is often clearer and easier to debug.
Q: Why model an agent as a graph or state machine instead of a while loop?
A: A bare loop hides the control flow, so limits, branches, and error paths get scattered through code. A graph makes nodes, transitions, and terminal conditions explicit, so you can enforce a max step count, unit test one node, log every transition, and checkpoint state to resume or pause for human approval. The cost is more up-front structure and a state object that must serialize cleanly.
Q: How do you decide retry and timeout settings for a step?
A: Work backwards from the user-facing deadline: attempts times per-attempt timeout plus backoff plus fallback time must fit. Retry only transient errors (timeouts, rate limits, 5xx) with exponential backoff and jitter, and never retry blindly on validation failures. With illustrative numbers, a 10 s timeout and 2 retries already reaches 30+ s before the fallback responds, so a tight deadline forces fewer retries or shorter timeouts.
Q: What would you trace in an LLM pipeline?
A: One trace per request with a span per step: step name, attempt number, latency, model and prompt version, input and output token counts, status and error, plus the retrieved document IDs for RAG steps. This lets you answer where time and cost went and which step produced a wrong answer. Be careful with user data: prompts and responses often contain sensitive content, so redact or sample what you store.
Q: How do you limit vendor or framework lock-in?
A: Put your own small interfaces (model client, retriever, tool registry) at the boundary and keep framework types out of your domain logic, keep prompts in a versioned store you own, and emit traces in a neutral format. That way replacing a provider or the framework means rewriting adapters, not the application. Some coupling is unavoidable and the abstraction itself may leak provider-specific features, so test the swap rather than assume it.
Q: Where do retrieval and orchestration meet?
A: Retrieval is one step in the graph: embed the query, search the index, and put the chunks into the prompt budget. Retrieval quality bounds answer quality, so evaluate that step separately from generation; an orchestration layer helps by tracing what was retrieved for each request. See RAG for chunking, ranking, and evaluation.
Follow-up: A chain that worked yesterday now returns worse answers after a dependency upgrade. How do you investigate?
A: Compare traces before and after, focusing on the final rendered prompt, since a library upgrade can change default templates, parser instructions, or message formatting without any change in your code. Pin versions, diff the exact request sent to the model, and re-run the golden set against both. This is the concrete argument for pinning dependencies and for being able to see the raw request.
Follow-up: How do you make a retried tool call safe?
A: Give each logical action an idempotency key generated once per run and step, and have the tool or downstream API deduplicate on it, so a retry or a resumed run returns the earlier result instead of repeating the side effect. Delivery to the tool is at-least-once; idempotency makes the effect effectively once. Read-only tools need no key, and irreversible actions should also require confirmation; see idempotency and exactly-once semantics.
Follow-up: A fallback model returns a lower-quality answer. Is that acceptable?
A: It depends on the product. A degraded answer is usually better than an error for low-stakes queries, but for high-stakes ones a clear "try again later" or human escalation is safer. Record that a fallback was used (the trace shows it), alert on the fallback rate, and evaluate the fallback path with the same golden set so you know its quality before an outage forces it on users.
In a RAG support-bot design, the orchestration layer is the service that owns the request lifecycle: authenticate, retrieve, build the prompt under a token budget, call the model with a timeout and retry policy, validate the output, fall back to a smaller model or a canned reply, and emit a trace. State your assumptions when estimating: at 1M requests/day, the average is about 12 requests/second (peak perhaps several times higher), and if each request makes 3 model calls, the layer coordinates roughly 3M calls/day, so a retry rate of even a few percent adds noticeable extra load and cost (illustrative arithmetic, not a measurement). The step deadline budget matters as much as throughput: a 30 s user-facing deadline split across retrieval, generation, and validation dictates the timeouts of each step. Multi-provider routing and rate limits usually sit in a gateway in front, see LLM gateway and multi-provider routing, while the orchestration layer stays focused on the workflow. Long-running agent tasks belong on a queue with checkpointed state rather than a held HTTP connection, and a human-approval node is just another state in the graph.
Adopting a framework buys speed to first result, connectors, a shared vocabulary, and sometimes built-in tracing and persistence; it costs transparency (what exactly was sent to the model), a moving dependency that can break on upgrade, and lock-in that grows as framework types spread through your code. Hand-rolling costs more up front and means you own retries, tracing, and integrations, but the code is small, direct, and easy to debug, and the failure modes are yours. The wrong choice is adopting a heavy framework for a two-step flow, or hand-writing a full agent runtime with checkpointing when a maintained graph library already does it. Graph-style control adds structure that pays off with loops and branches and is overhead for a linear chain. Whatever you pick, keep a thin boundary of your own, pin versions, insist on seeing the raw prompt and response, and revisit the decision once the pipeline stabilizes and performance and cost matter more than iteration speed.