An intern with a to-do list and a set of office tools. They think about the next step, use a tool (search, email, spreadsheet), read the result, and repeat until done, while a manager must sign off on anything risky like refunds, and a diary records every step.
Problem Statement
Design a platform for AI agents: an LLM that doesn't just answer but takes actions by calling tools (search the web, query a database, send an email, run code) in a loop until a task is done. Examples: coding agents, customer-support agents that can issue refunds, research assistants. The system must run long, multi-step tasks reliably and safely.
Functional Requirements
Start an agent run from a user request; the agent plans, calls tools, reads results and continues until it answers or gives up.
A registry of tools with schemas and permissions; per-customer tool configuration.
Human approval for sensitive actions (payments, emails, deletions).
Stream progress to the user; show a trace of every step.
Out of scope: training the model; building each individual tool.
Non-Functional Requirements
Reliability of long runs: runs can take minutes and dozens of steps; a crash must not lose progress or repeat side effects.
Safety: least privilege, sandboxed code execution, prompt-injection defence, audit trail.
Cost and loop control: step, token and time budgets per run.
Scale: e.g. 100K concurrent runs, each mostly waiting on LLM or tool I/O.
Core Entities
Run: id, user, goal, status (running/waiting_approval/succeeded/failed), budgets, created_at.
Tool: name, JSON schema, permissions, rate limits, whether it has side effects.
Memory: per-user or per-project facts and documents available to retrieve.
API Design
POST /v1/runs { "goal": "Refund order 812 if it arrived damaged", "agent": "support" } -> { run_id }
GET /v1/runs/{id}/events (SSE: step started, tool called, approval needed, done)
POST /v1/runs/{id}/approvals/{step} { "decision": "approve" | "reject" }
POST /v1/runs/{id}/cancel
High-Level Design (HLD)
The orchestrator asks the model for the next step, runs tools safely, pauses for approval when needed, and logs everything.
Orchestrator runs the agent loop as a durable workflow: every step is persisted before and after it executes, so a crashed worker resumes from the last completed step (Temporal-style).
LLM gateway calls models with the conversation so far plus tool schemas; the model replies with text or a structured tool call.
Tool executor validates arguments against the schema, checks permissions, calls the tool (HTTP APIs, or code in an isolated sandbox), and returns a size-limited result.
Approval service pauses the run on sensitive tools until a human decides.
Trace store keeps every step for debugging, evaluation and audit.
Low-Level Design (LLD)
The step log is what makes runs resumable, auditable and replayable for evaluation.
The agent loop
while run.steps < budget.max_steps and not run.done:
reply = llm(messages + tool_schemas) # persisted as a step
if reply.is_final_answer:
finish(run, reply.text); break
for call in reply.tool_calls:
validate(call.args, tool.schema); authorize(run.user, tool)
if tool.sensitive: wait_for_approval(run, call) # run status = waiting
result = execute(tool, call, idempotency_key=f"{run.id}:{step}")
messages.append(tool_result(call.id, truncate(result))) # persisted
if budget exceeded: fail(run, "step/token budget exhausted")
Reliability
Durable steps: a worker crash replays the run from its log without re-executing completed tool calls (their results are recorded).
Tool calls with side effects carry an idempotency key derived from run id + step, so a retried step can't double-refund.
Timeouts per tool call; retries with backoff for transient errors; the model is told about errors so it can adapt.
Safety
Treat tool output (web pages, emails, documents) as untrusted data: it can contain prompt injection. Never let it grant new permissions; keep sensitive actions behind approval or strict policy checks.
Least privilege: tools act with the user's own scoped credentials, never an all-powerful service account.
Code runs in sandboxes (containers/microVMs) with no network or a strict allowlist, CPU/memory/time limits, and no secrets.
Follow-up Questions
The agent gets stuck repeating the same tool call. How do you stop it?
Budgets (max steps, tokens, wall time), loop detection (same call with same arguments N times), and returning a clear error so the model changes approach; if still stuck, end the run and hand off to a human.
How do you evaluate agent quality?
Replay a suite of recorded tasks against new prompts/models and score outcomes (task success, cost, steps, policy violations); traces from production feed new test cases.
Why a durable workflow engine instead of a simple loop in a web request?
Runs outlive requests (minutes, waiting on humans), must survive deploys and crashes, and must not repeat side effects. A durable log of steps provides all three.