Observability is the ability to answer "what is happening inside this system, and why" from the telemetry it emits (metrics, logs and traces) without shipping new code to find out. AI systems make that harder than a typical CRUD backend because the work per request varies enormously (a 20-token answer and a 2,000-token answer are the same endpoint), latency has several components with different causes (queueing, retrieval, prefill, decode), cost scales with tokens rather than requests, and "the answer was wrong" is a failure that no HTTP status code reports. Interviewers probe it to see whether you can debug a slow or expensive request end to end, choose signals that do not lie (percentiles, not averages), and avoid the classic self-inflicted problems: exploding metric cardinality and prompts full of personal data landing in a log index.
Observability is the ability to understand what's happening inside a running system, and why something went wrong or got slow, just by looking at the information it produces automatically, like counters, logs and step-by-step traces of a request, without having to add new code and redeploy just to investigate. For AI systems specifically, this means being able to see not just whether a request succeeded, but how long each stage took, how much it cost, and whether the answer that came back was actually any good.
Think of a hospital's patient monitoring equipment. Doctors don't have to open up a patient to know something is wrong; heart rate, oxygen levels and blood pressure are tracked continuously on a monitor, and an alarm sounds the moment something crosses a dangerous threshold. Observability does the same job for software: it continuously tracks vital signs like response time, error rate and cost, so engineers can spot and diagnose problems without cracking open the system itself.
Pillar Answers Cost profile Example for an LLM endpoint
metrics "how much / how often / how cheap, aggregated, fixed size requests_total{route,status}, latency histogram,
slow, right now and trending" per series tokens_total{direction}, queue_depth
logs "what happened in THIS event" grows with volume; searchable structured JSON: trace_id, model, tokens, error
traces "where did the time go across per-request; usually sampled span tree: gateway -> retrieve -> rerank -> llm
services for ONE request"
RED (for request-driven services) USE (for resources)
Rate requests/sec Utilization % of capacity busy (GPU util, KV-cache % full)
Errors failed requests/sec Saturation queue length / waiting requests
Duration latency distribution Errors device or OOM errorsThe three pillars answer different questions and you need all of them: a metric tells you that p99 doubled at 14:05, a trace tells you which stage of one slow request ate the time, and a log line tells you the specific error string. RED (rate, errors, duration) suits anything a user calls; USE (utilization, saturation, errors) suits the resources behind it, and for LLM serving the saturation signals are the ones that predict pain first: requests waiting in the queue, and KV-cache occupancy. Exact metric names differ by serving stack (vLLM, Triton, TGI and managed APIs each expose their own), so say "depending on the framework" and name the concept.
AI adds signals that the classic dashboards miss: input and output tokens per request, time to first token, inter-token latency, queue wait separated from model compute, cache hit rate (prompt or semantic cache), retry and fallback counts across providers, and a quality signal such as thumbs-down rate or automated evaluation scores. Without the queue-wait versus compute split, "the model got slow" and "we ran out of GPU capacity" look identical on a latency graph but need opposite responses. For the gateway view of the same data see Design an LLM API Gateway.
One trace = one user request; each stage is a span with start time, duration and attributes.
POST /chat (root span) trace_id = 4bf92f3577b34da6a3ce929d0e0e4736
|-- auth+rate_limit
|-- embed_query attrs: tokens_in
|-- retrieve
| |-- ann_search attrs: k=20
| `-- rerank attrs: candidates=20
|-- queue_wait <- time waiting for a GPU slot, NOT model compute
`-- llm_generate attrs: model, tokens_in, tokens_out, ttft_ms
|-- prefill_to_first_token
`-- decode
Context propagation (W3C Trace Context): traceparent: 00-<trace-id>-<parent-span-id>-01
Every hop (HTTP, gRPC, queue message headers) forwards it; a hop that drops it breaks the tree in two.A trace is a tree of spans sharing one trace id, and the parent-child structure is what answers "which stage is responsible?". In a RAG request the interesting boundaries are embedding, ANN search, reranking, queue wait and generation; in an agent the tree gets a span per reasoning step and per tool call, and the total time is dominated by however many loops the agent took, which is exactly the number you want to see. OpenTelemetry is the common instrumentation standard, with the trace id carried between services in the W3C traceparent header (and in message headers when a queue sits in the middle, since context does not cross async boundaries by itself). OpenTelemetry has been developing semantic conventions for generative-AI spans (model name, token usage, and so on), but they have been evolving, so check the current spec rather than quoting attribute names from memory. The wire-level and collector side of tracing is covered in Distributed Tracing & Observability.
The failure modes at scale are practical. Tracing every request is expensive, so you sample, and head-based sampling (decide at the start, for example 1%) is cheap but discards the rare slow or failed request you most want, while tail-based sampling (decide after the trace completes, keeping errors and slow ones) needs a collector that buffers whole traces. Missing propagation through a queue or a thread pool produces orphan spans that look like separate short requests. And spans that simply do not add up (a root of 1,800 ms with 900 ms of children) point at uninstrumented work, which is itself a finding.
Request timeline for a streamed LLM response:
t0 request in ---- queue ---- prefill ----> FIRST TOKEN ---- decode, one token at a time ----> last token
|<----- TTFT -----|<-------- inter-token latency x N --------|
|<-------------------- total latency --------------------------|
TTFT ~ queue wait + network + prefill (grows with input length) what the user perceives as "it started"
decode time ~ output tokens x per-token latency what the user perceives as "typing speed"
Cost per request (ILLUSTRATIVE arithmetic; use your provider's current price sheet):
cost = tokens_in x price_in + tokens_out x price_out
1M requests/day x 1,500 in x 300 out = 1.5B input tokens + 0.3B output tokens per day
A 20% drop in prompt length is a 20% drop in the input side; a retry storm doubles BOTH sides.Mean latency hides the users who suffer. In the demo below, a sample where 3% of requests are stragglers (cold caches, long generations, a retry) has a mean near 530 ms and a median near 410 ms, so the average looks healthy, while p99 is around 4.6 s: roughly 1 in 30 requests is about ten times slower than the typical one. At 1,000 requests per second that is 30 slow requests every second. Percentiles expose that; and for a user session that makes many calls the chance of at least one slow call is far higher than the per-request percentile suggests. Track TTFT and total latency as separate distributions for streamed endpoints (see Streaming Responses), because a fast first token with a slow tail feels very different from a slow start.
Two rules keep percentiles honest. Store histograms, not pre-computed percentiles: p99 from shard A and p99 from shard B cannot be averaged into a global p99, but bucket counts add together, and the price is precision limited by bucket width (the demo shows a histogram estimate of roughly 4.65 s against an exact 4.58 s). And slice by the right dimensions: input-token bucket, model, route and cache hit versus miss, because latency in LLM serving correlates strongly with token counts, so a global p99 mostly reports "some requests were long". For cost, attribute tokens to a tenant, feature and model so you can answer "who spent it" (see Cost Optimization for AI Systems).
Cardinality = number of distinct label combinations = number of time series stored.
model(5) x route(20) x status(5) = 500 series fine
x tenant(1,000) = 500,000 series expensive
x histogram buckets (~10 per series) = 5,000,000 series a real bill / an outage of the metrics backend
x user_id or prompt_hash (unbounded) = grows forever NEVER a metric label
Put high-cardinality identifiers in LOGS and TRACE ATTRIBUTES (searched per event), not metric labels.
Logging prompts safely
log by default: ids, model, token counts, latencies, status, prompt_template_version, prompt LENGTH
do not log by default: raw prompts/completions (PII, secrets, customer data)
if you must: redact (regex/NER), hash or tokenise identifiers, sample a small fraction into a
restricted-access store with a short retention window, and follow your data-protection obligations.Cardinality is the number one way observability systems fall over, because every distinct label combination is a separate stored series and the count multiplies across labels. The rule is that metric labels must have a small, bounded set of values; identifiers such as user id, request id or prompt hash belong on a log line or a trace attribute, where the cost is per event rather than per combination. Histogram buckets multiply the series count again, so choose buckets for the latencies you care about instead of adding more labels.
Prompts and completions are the sensitive part: users paste emails, credentials and health details into chat boxes, and retrieved documents carry tenant data. Treat raw text as a liability. Log structure by default, redact before writing, and if you keep samples for debugging or evaluation, keep them in a restricted store with short retention and access control (whether and how long you may retain them is a policy and legal question, not just an engineering one). Evaluation traces close the loop: attach to a trace the retrieved chunk ids and scores, the prompt template version, the model version and the output, then run offline or sampled online evaluations (groundedness, relevance, a judge model, user feedback) against those traces. When quality drops, you can tell whether retrieval brought the wrong chunks or the model ignored the right ones, and you can replay the same inputs against a new prompt or model. This connects to Model Monitoring & Drift Detection and to the retrieval-versus-generation split in Retrieval-Augmented Generation.
SLI = a measured ratio of good events / valid events SLO = the target for it over a window
availability SLI : non-5xx responses / all valid requests
latency SLI : requests with TTFT under a threshold / all requests (a threshold, not an average)
quality SLI : sampled answers passing an evaluation / sampled answers (noisier, slower to measure)
Error budget = 1 - SLO. 99.9% over 30 days: 0.001 x 43,200 min = 43.2 minutes of "badness"
Burn rate = (observed error rate) / (1 - SLO). 1.0 means the budget is spent exactly at the end of the window.
burn 14.4 for 1 hour -> 14.4 x 1h / 720h = 2% of the monthly budget gone in an hour -> page someone
burn 6 for 6 hours -> 6 x 6 / 720 = 5% of the budget -> page or urgent ticket
burn 1 for 3 days -> a slow leak that exhausts the budget right on schedule -> ticket
Multi-window: require BOTH a long window (is it real?) AND a short one (is it still happening?) to fire.Alerting on raw thresholds ("error rate above 1% for 5 minutes") pages people for blips that consume no meaningful budget and misses slow bleeds that do. An SLO turns the question into "how fast are we spending the error budget?": the burn rate is the current error rate divided by the allowed error rate, and alerting on burn rate over paired long and short windows pages quickly for a severe outage and stays quiet for a brief spike. The numbers above are the commonly cited examples from Google's SRE workbook; treat them as a starting point to tune, not a standard.
AI services need care choosing the SLI. Latency SLIs should be on TTFT and on total latency for a given output-length bucket (otherwise long answers "violate" the SLO by design). Quality does not fit an availability-style ratio cleanly: sampled evaluation scores are noisy and lag, so they are better as a tracked SLI with a slower alert than as a pager. And the dependency on an external provider matters: if you fall back to another model when the first times out, the user-facing SLI can stay green while cost and quality shift, so instrument the fallback rate explicitly. Related capacity and target-setting material is in Capacity Planning & SLOs.
First, a seeded latency sample with a 3% straggler tail: mean versus percentiles, the share of slow requests, and a Prometheus-style bucketed p99 estimate. The distribution is synthetic, generated for the demo; it is not a measurement of any real model or provider.
import random, statistics
random.seed(7)
# Seeded, skewed latency sample (ms): mostly fast, ~3% cold-cache / long-generation stragglers.
lat = []
for _ in range(10_000):
if random.random() < 0.03:
lat.append(random.uniform(2000, 6000)) # straggler
else:
lat.append(random.lognormvariate(6.0, 0.35)) # typical: median around 400 ms
def pct(sorted_vals, p): # nearest-rank percentile
k = max(0, min(len(sorted_vals) - 1, int(round(p / 100 * len(sorted_vals) + 0.5)) - 1))
return sorted_vals[k]
s = sorted(lat)
print(f"n={len(s)} mean={statistics.mean(s):7.1f} ms")
for p in (50, 95, 99, 99.9):
print(f"p{p:<5}= {pct(s, p):7.1f} ms")
print(f"max = {s[-1]:7.1f} ms")
slow = sum(1 for x in s if x > 1500)
print(f"requests slower than 1500 ms: {slow} ({slow / len(s):.1%})")
# Prometheus-style histogram: only bucket counts are stored, p99 is ESTIMATED by interpolation inside a bucket.
bounds = [100, 250, 500, 1000, 2500, 5000, 10000]
counts = [0] * len(bounds)
for x in lat:
for i, ub in enumerate(bounds):
if x <= ub:
counts[i] += 1; break
def hist_quantile(q):
rank, cum, lo = q * len(lat), 0, 0
for ub, c in zip(bounds, counts):
if cum + c >= rank:
return lo + (ub - lo) * (rank - cum) / c
cum, lo = cum + c, ub
print("bucket counts:", dict(zip(bounds, counts)))
print(f"histogram-estimated p99 = {hist_quantile(0.99):7.1f} ms | exact p99 = {pct(s, 99):7.1f} ms")
# Output:
# n=10000 mean= 528.2 ms
# p50 = 408.2 ms
# p95 = 805.4 ms
# p99 = 4578.9 ms
# p99.9 = 5803.9 ms
# max = 5958.9 ms
# requests slower than 1500 ms: 287 (2.9%)
# bucket counts: {100: 0, 250: 838, 500: 6295, 1000: 2538, 2500: 82, 5000: 171, 10000: 76}
# histogram-estimated p99 = 4649.1 ms | exact p99 = 4578.9 msSecond, a toy span tree for one RAG request. The stage durations are invented illustrative milliseconds (including the 5 ms per output token), chosen only to show how a trace attributes time to stages and how TTFT falls out of the tree.
import random
random.seed(11)
class Span:
def __init__(self, name, start, dur, attrs=None):
self.name, self.start, self.dur, self.attrs, self.children = name, start, dur, attrs or {}, []
def build_request():
"""Simulated RAG request. Times are made-up milliseconds, not measurements of any real system."""
root = Span("POST /chat", 0, 0)
t = 0
def add(parent, name, dur, **attrs):
nonlocal t
sp = Span(name, t, dur, attrs); parent.children.append(sp); t += dur; return sp
add(root, "auth+rate_limit", random.randint(3, 8))
add(root, "embed_query", random.randint(20, 40), tokens_in=18)
r = add(root, "retrieve", 0)
st = t
add(r, "ann_search", random.randint(15, 30), k=20)
add(r, "rerank", random.randint(40, 90), candidates=20)
r.dur = t - st
add(root, "queue_wait", random.randint(30, 200))
g = add(root, "llm_generate", 0)
st = t
ttft = random.randint(250, 450)
add(g, "prefill_to_first_token", ttft, tokens_in=1450)
add(g, "decode", 220 * 5, tokens_out=220) # illustrative: 5 ms per output token
g.dur = t - st
g.attrs.update(ttft_ms=ttft, tokens_in=1450, tokens_out=220)
root.dur = t
return root
def show(sp, total, depth=0):
bar = "#" * max(1, round(sp.dur / total * 40))
print(f"{' ' * depth}{sp.name:<{26 - 2 * depth}} {sp.start:5d} +{sp.dur:4d} ms {bar}")
for c in sp.children:
show(c, total, depth + 1)
root = build_request()
show(root, root.dur)
print()
g = root.children[-1]
child_sum = sum(c.dur for c in root.children)
print(f"root={root.dur} ms, sum(children)={child_sum} ms (no unexplained gap)")
slowest = max(root.children, key=lambda c: c.dur)
print(f"slowest stage: {slowest.name} = {slowest.dur / root.dur:.0%} of the request")
print(f"TTFT ~= time before first token = {g.start + g.attrs['ttft_ms']} ms; decode = {g.children[1].dur} ms for {g.attrs['tokens_out']} tokens")
# Output:
# POST /chat 0 +1800 ms ########################################
# auth+rate_limit 0 + 6 ms #
# embed_query 6 + 37 ms #
# retrieve 43 + 97 ms ##
# ann_search 43 + 29 ms #
# rerank 72 + 68 ms ##
# queue_wait 140 + 160 ms ####
# llm_generate 300 +1500 ms #################################
# prefill_to_first_token 300 + 400 ms #########
# decode 700 +1100 ms ########################
#
# root=1800 ms, sum(children)=1800 ms (no unexplained gap)
# slowest stage: llm_generate = 83% of the request
# TTFT ~= time before first token = 700 ms; decode = 1100 ms for 220 tokensNot runnable here (no output invented): with OpenTelemetry you would wrap each stage in tracer.start_as_current_span("retrieve") and export through an OTLP collector; check the current SDK docs for the API and the generative-AI semantic conventions.
Q: What is the difference between monitoring and observability?
A: Monitoring checks known failure conditions with dashboards and alerts; observability is having enough rich, correlated telemetry to investigate failures you did not anticipate. In practice that means high-detail traces and structured logs you can slice by arbitrary attributes, on top of the metrics that drive alerts. The trade-off is cost: more detail means more storage, so you sample and bound cardinality.
Q: Why are percentiles better than averages for latency?
A: Latency distributions are skewed, so the mean is pulled around by a small tail and describes almost nobody's actual experience. In the demo above the mean is about 530 ms and p99 is about 4.6 s, so an average-based alert sees nothing while roughly 3% of users wait many seconds. Report p50, p95 and p99 (and be explicit about the window), computed from histograms.
Q: A user says the chatbot is slow. How do you debug it?
A: Find the request by id or user and open its trace, then look at which span dominates: queue wait points at capacity or a noisy neighbour, retrieval or rerank points at the vector store or a cold index, prefill points at a long prompt (check tokens_in), decode points at a long output or a slow GPU. If the trace is missing or has a gap, that is uninstrumented work. Then check whether the aggregate metrics show the same pattern for other requests, so you know if it is one bad request or a regression.
Q: What is TTFT and why track it separately from total latency?
A: Time to first token is the delay before the first output token arrives; it is roughly queue wait plus network plus the prefill phase, which grows with prompt length. Total latency additionally includes decode, which scales with the number of output tokens. For streaming UIs users judge responsiveness mostly by TTFT and reading speed by inter-token latency, so an SLO on total latency alone hides a slow start behind a short answer.
Q: What is metric cardinality and why does it matter for AI systems?
A: It is the number of distinct label combinations, each stored as its own time series. Adding a label with many values (tenant, user id, prompt hash) multiplies the series count and can overwhelm the metrics backend and its bill. Keep metric labels bounded (model, route, status) and put high-cardinality ids on logs and trace attributes.
Q: Can you log full prompts for debugging?
A: By default no: prompts and completions often contain personal data, secrets and customer documents, and log systems are widely readable and long-lived. Log ids, token counts, template and model versions, and lengths; if raw text is needed, redact, sample a small fraction into a restricted store with short retention, and involve your privacy and legal requirements. The cost is harder debugging of content-specific bugs, which evaluation datasets and opt-in capture can partly offset.
Q: What SLIs would you set for an LLM chat endpoint?
A: Availability (non-error responses over valid requests), a TTFT latency SLI defined as the fraction under a threshold, and a total-latency SLI bucketed by output length. Add a quality indicator from sampled evaluations or user feedback, tracked with a slower alert since it is noisy. Each SLO gets an error budget and burn-rate alerts rather than raw threshold alerts.
Follow-up: You have 20 replicas and each reports its own p99. How do you get the fleet p99?
A: You cannot average or take the max of the per-replica percentiles and get the true value. Export histograms with identical bucket boundaries, sum the bucket counts across replicas, and compute the quantile from the merged histogram; the error is bounded by bucket width. If you need higher precision, use finer buckets around the SLO threshold or a mergeable sketch such as t-digest or DDSketch where your backend supports it.
Follow-up: Explain multi-window burn-rate alerting to someone used to "error rate above X".
A: The burn rate is the current error rate divided by the error rate the SLO allows, so 1 means you will exactly exhaust the budget by the end of the window. You alert when a fast burn (for example a high multiple over one hour) is confirmed by a short recent window, so real outages page quickly and already-recovered blips do not, and you use a lower multiple over longer windows for slow leaks that only need a ticket. It ties alerting to user-visible budget consumption instead of arbitrary thresholds.
Follow-up: Tracing is costing too much. What do you change without going blind?
A: Switch to tail-based sampling so you keep 100% of errors and slow traces and a small fraction of healthy ones, trim high-volume low-value spans and large attributes, and keep metrics unsampled since they are cheap and drive alerts. Head sampling alone at a low rate would drop precisely the rare bad requests. The trade-off is that tail sampling needs a collector that buffers full traces, which adds memory cost and a component to operate.
In an LLM gateway or RAG platform design, observability is the layer that makes SLOs, autoscaling and cost control possible: the gateway assigns the trace id, records tokens and cost per tenant, and exports TTFT and latency histograms, while the serving layer exports queue depth and KV-cache use that autoscalers and capacity plans consume. A useful estimation for the interviewer: assume 100 requests per second, 10 spans per request and 500 bytes per span; full tracing is then 100 x 10 x 500 = 500 KB/s, about 43 GB per day, and a 5% sample is about 2 GB per day (ILLUSTRATIVE, before compression and indexing overhead), which is why sampling policy is a real design decision. Metric volume is governed by series count (labels multiplied together) rather than request rate, so cardinality budgeting belongs in the design. Alerting rides on SLOs with burn-rate rules, evaluation traces feed drift and quality monitoring, and retention for prompt text is set jointly with privacy requirements rather than defaulting to "keep everything".
Observability trades money, engineering time and privacy risk for the ability to debug in production. Richer telemetry (full traces, high-cardinality tags, raw prompts) makes investigations faster but multiplies storage cost, can overload the metrics backend, and creates a store of sensitive data that must itself be secured, so the practical design is layered: cheap unsampled metrics for alerting, sampled and tail-biased traces for diagnosis, structured logs without raw content by default, and opt-in or sampled capture of text under controls. It is the wrong investment to instrument everything at maximum detail on day one, and equally wrong to ship an LLM service with only request count and mean latency, because the failures that hurt (queue saturation, long-tail latency, token-cost blowups, silent quality regressions) will not show up there. Start from SLOs and the questions you expect to ask during an incident, and add detail where those questions cannot be answered.