Distributed Tracing & Network Observability

A single user request to a modern backend might cross a dozen microservices, a database, a cache, and a third-party API. When it's slow or fails, per-service logs alone can't tell you where — you need to follow that one request across every network hop it made. This is the problem distributed tracing solves, and it's a near-guaranteed topic when a senior interview touches production debugging.

What Is Distributed Tracing?

Distributed tracing is a way of following one single request as it travels through every separate service that helps handle it, so you can see the whole journey as one connected timeline instead of a pile of separate, disconnected logs. Each service adds its own timestamped piece to that timeline, and all the pieces get stitched back together under one shared ID.

What Is Network Observability?

Observability is the broader idea that a system's internal health should be understandable from the outside, using the signals it produces — logs (what happened), metrics (how much, how often, how fast), and traces (the path a single request took). A system is "observable" when an engineer can answer a new, unplanned question about it (like "why did this one request take three seconds?") without having to ship new code first.

Where This Is Used

• Engineering teams at companies like Uber, Netflix, and Datadog customers rely on tracing dashboards to debug slow or failed requests in production.
• Tools like Jaeger, Honeycomb, New Relic, and Datadog APM are built specifically to collect and visualize these traces.
• On-call engineers use trace waterfalls during an incident to find which one of many services is actually responsible for an outage.
• Any app that talks to a third-party API (a payment processor, an LLM provider) uses tracing to prove whether a slowdown is their own code or the vendor's.

A Day-to-Day Example

Imagine tracking a package that passes through several delivery hubs before reaching your door. Each hub scans it and logs a timestamp, but if you only look at one hub's logbook, you have no idea how long the package sat at the hub before that, or which hub caused the delay. A tracking number is what lets you pull every hub's scan together into one timeline and see exactly where the package slowed down — that tracking number is exactly what a trace ID does for a request moving through a company's services.

Deep Dive: Anatomy of a Trace

Take a request to an AI-backed API that takes 2 seconds end-to-end. Without tracing, all you know is "2 seconds, slow." A trace breaks it into spans — one per unit of work, each with its own start time and duration, nested to reflect the call graph:

Trace: GET /api/summarize   (trace_id: 4bf92f...)  total: 2005ms

  [root span] api-gateway                          0ms  →  2005ms
    ├─ span: DNS lookup (internal service)          0ms  →    10ms
    ├─ span: TCP+TLS to backend                    10ms  →    30ms
    ├─ span: backend-service handler               30ms  →  2000ms
    │    ├─ span: redis GET cache:summary:123      30ms  →    35ms   (cache miss)
    │    ├─ span: postgres SELECT documents        35ms  →    85ms
    │    └─ span: llm-client.Complete()            85ms  →  1885ms   ← 90% of total
    └─ span: response serialize + return         2000ms  →  2005ms

That waterfall immediately tells you the LLM call, not the infrastructure around it, dominates latency — 1800ms of 2005ms. Without a trace, you might waste time optimizing Redis or the database when they're contributing 55ms combined. This is the entire value proposition of tracing in one example: it turns "it's slow" into "this specific span, in this specific service, is slow."

Deep Dive: Trace Propagation — How Spans From Different Services Get Stitched Together

A trace is only useful if every service along the path agrees it's part of the same trace. That's done by propagating a trace context in request headers, standardized as W3C Trace Context's traceparent header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                │
             │  └─ trace-id (32 hex chars,        │                └─ flags
             │     shared by every span in         │                   (01 = sampled)
             │     this entire trace)               └─ parent-id (this
             version                                   span's ID, becomes
                                                         the child's parent
                                                         reference)

When api-gateway calls backend-service, it sends its own span ID as the parent-id in the outgoing traceparent header. backend-service reads that header, generates its own new span ID (with the same trace-id), and records "my parent is api-gateway's span." When backend-service later calls the LLM API, it does the same thing again. The tracing backend later reconstructs the whole tree purely from these trace-id/parent-id relationships across every span it received, regardless of which service or host produced each one. Miss propagating this header across even one hop — a common bug at an async boundary like a message queue — and the trace has a gap: the downstream work becomes an orphaned, disconnected trace instead of a child span.

Key Pointers

• A trace represents one end-to-end request; it's composed of spans, where each span is a single unit of work (an HTTP call, a DB query, a function) with its own start time and duration.
• A trace ID is generated at the edge (or first service) and propagated through every downstream call, usually in HTTP headers (traceparent in the W3C Trace Context standard); each hop adds a span ID and references its parent span, building a tree.
• This differs from logging (discrete, timestamped events per service, no inherent cross-service linkage) and metrics (aggregated numeric time series, e.g. request rate or p99 latency) — tracing, logs, and metrics are the three complementary "pillars of observability."
• OpenTelemetry (OTel) is the vendor-neutral standard for instrumenting code to emit traces/metrics/logs; Jaeger and Zipkin are backends that collect, store, and visualize those traces as waterfall diagrams.
• Propagation has to survive every hop, including async ones (message queues) — dropping the trace context anywhere breaks the chain and leaves a gap in the trace.
• Sampling matters at scale: tracing every single request is expensive, so production systems typically sample (e.g. 1% of traces, or all traces above a latency threshold) rather than capture everything.

How It Works

1. A request arrives at the entry point (API gateway or edge service); a tracing library generates a new trace ID and a root span ID.
2. The trace ID and current span ID are injected into outgoing request headers whenever this service calls another service.
3. Each downstream service extracts the trace ID from incoming headers, starts its own child span linked to the parent, and does the same propagation for any calls it makes.
4. Every service exports its finished spans (asynchronously, so it doesn't block the request) to a collector — often via the OpenTelemetry Collector.
5. The tracing backend (Jaeger/Zipkin/a vendor APM) stitches spans with the same trace ID back into a single waterfall view, showing exactly which hop added the latency or threw the error.

Try It Yourself

You can see propagation with nothing more than curl, and inspect real traces with a local Jaeger instance:

# Send a request carrying an explicit trace context — many OTel-instrumented
# services will pick this up and continue the same trace
curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
  https://api.example.com/orders/123

# Run Jaeger locally (all-in-one image includes UI + collector + storage)
docker run -d --name jaeger \
  -p 16686:16686 -p 4317:4317 -p 4318:4318 \
  jaegertracing/all-in-one:latest

# Point an OpenTelemetry-instrumented app at it via env vars, then open
# the UI to search traces by service, operation, or trace ID:
#   http://localhost:16686
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_SERVICE_NAME=backend-service

# In the Jaeger UI: pick a service → find a slow trace → the waterfall
# view shows every span, its duration, and which one ate the most time —
# exactly the redis/postgres/llm-client breakdown shown above.

Interview Q&A

Q: A production API is slow. Walk me through how you'd find the bottleneck.

A: Start with metrics to confirm it's real and see scope — is p99 latency up across the board, or is it one endpoint? Then pull a trace (or a sample of traces) for slow requests specifically and read the waterfall: DNS time, TCP/TLS setup, time-to-first-byte from the load balancer, each downstream span (cache, DB, any external API/LLM call). Whichever span accounts for most of the total duration is the bottleneck — often it's an external call, not your own code. Confirm with logs from that specific service/span if you need more detail than the span itself carries (e.g. a specific slow query). This is the metrics-narrow-with-a-trace-confirm-with-logs loop.

Q: What's the difference between a trace ID and a span ID?

A: The trace ID is shared by every span belonging to one end-to-end request — it's what lets a backend stitch spans from different services back together. A span ID identifies one specific unit of work within that trace; each span also carries a reference to its parent span ID, which is what lets the backend reconstruct the call tree, not just a flat list of spans.

Q: Why sample traces instead of capturing every request?

A: At high request volume, instrumenting and exporting a span for every hop of every request adds real CPU/network overhead and storage cost that scales linearly with traffic. Most systems sample — e.g. 1% of all requests, plus a rule that always captures anything above a latency threshold or with an error — trading complete coverage for a representative (and much cheaper) picture, while still guaranteeing you capture the traces you'd actually want to look at.

Q: What breaks tracing across a message queue?

A: The trace context (trace ID + parent span ID) has to be explicitly attached to the message itself, usually as message metadata/headers, when it's published — and the consumer has to read it back out and continue the same trace when it processes the message. If either side forgets, the async work still happens, but shows up as a disconnected, orphaned trace instead of a child span of the original request, and you lose the ability to see it as part of the same request's timeline.

Q: How are traces different from logs and metrics — give a concrete scenario.

A: An alert fires because p99 latency crossed a threshold — that's a metric, telling you something's wrong and how bad. You pull a sample of slow traces to see where, in the call graph, the time is actually going — that's tracing. Once you know it's e.g. the Postgres span, you pull that service's logs around that timestamp to see the actual slow query or error detail — that's logging. Each pillar answers a different question, and production debugging usually moves through all three in that order.

System Design Tie-in

"Design an observability platform" (or being asked how you'd add observability to a system you just designed) is really asking whether you can wire instrumentation into an architecture, not just recite OTel/Jaeger as buzzwords. A strong answer: every service is instrumented with an OpenTelemetry SDK; a trace ID is minted at the edge (API gateway) and propagated via traceparent through every synchronous call and attached explicitly to any message published to a queue; spans export asynchronously to an OpenTelemetry Collector, which batches and forwards to a backend (Jaeger, or a hosted APM); sampling is tuned to capture 100% of errors and slow requests but a small percentage of normal traffic to control cost; and metrics/logs correlate back to the same trace ID so an on-call engineer can pivot from a latency alert straight to the relevant trace to the relevant log lines without re-deriving the connection by hand.

Trade-offs

Tracing adds real overhead — instrumentation code in the hot path, network calls to export spans, and storage/query cost for the backend — so most orgs sample rather than trace every request, which means rare issues can slip through unsampled. It also requires discipline: every service and every async boundary (queues, background jobs) has to propagate context correctly, or the trace has holes. A common interview gotcha is treating logs, metrics, and traces as interchangeable — metrics tell you something is wrong and roughly how bad; logs tell you what a specific service saw; traces tell you where in the call graph the time or error actually happened. Production debugging usually starts with a metric alert, narrows with a trace, and confirms with logs.