LLM Observability in Production: Tracing, Evals, and Drift — A 2026 Reference Guide
Here is a scenario every LLM platform team eventually lives through. All mechanical dashboards are green: p95 latency fine, error rate flat, uptime 100 percent. Support tickets climb anyway. Users say the assistant "got worse" two weeks ago. Nothing in your infrastructure changed — but a retrieval index quietly drifted, a vendor model got updated under you, or a prompt tweak landed without an eval gate. The system is healthy and wrong at the same time.
Traditional observability cannot see this failure class, because the failure is in output quality, not in mechanics. LLM observability is the discipline built to see it. This guide covers what we instrument, what we ignore, and how the pieces fit together — the reference we hand to new engineering teams on their first agent deployment.
Why LLM systems need their own observability stack
Four properties of LLM applications break traditional APM assumptions:
1. Non-determinism. The same input can produce different outputs run to run. "Golden" replay testing does not directly apply; you evaluate distributions, not points. 2. Silent quality failure. A degraded system still returns 200s with fluent text. Quality has no built-in error code. 3. Composite execution. One user task fans out into many model calls, tool calls, and retrieval steps. Cost and latency live at the task level; debugging lives at the step level. You need traces that connect both. 4. External dependencies that change without notice. Vendor models get deprecated and updated; upstream data shifts. Your system's behavior can change with zero deploys by you.
Each property dictates a piece of the stack: traces for composite execution, evals for quality, drift monitoring for external change.
Layer 1: Tracing — the unit of truth
A trace is the complete record of one task: the input, every model call with its prompt and completion, every tool call with arguments and results, retrieval scores, token counts, latency, and cost per step, joined as a tree.
The moment you adopt agents, tracing stops being optional. A task that spans eight model calls and four tool calls is undebuggable from logs alone — you need the causal chain of what the model saw at each step.
What to capture per span:
| Field | Why |
|---|---|
| Input/output text (redacted) | Forensics: what did the model actually see and say |
| Token counts + cost | FinOps attribution per feature and per customer |
| Latency, start/end timestamps | SLO analysis, bottleneck location |
| Model/prompt version, temperature | Reproducibility: behavior changes traceable to config |
| Tool name, args (redacted), result status | Tool failure and misuse analysis |
| Trace ID propagated across steps | Reconstruct multi-step reasoning chains |
Sampling strategy. Traces are bulky. We keep full traces for a random sample of production traffic (the percentage your volume and privacy posture allow), plus 100 percent of traces that are escalated, appealed, high-cost, or low-confidence. Everything else gets metrics-only: counts, tokens, cost, latency histograms — no content.
Layer 2: Evals — quality as a measured quantity
An eval is a test with judgment instead of an assertion: given this input (and context), is this output correct? Three tiers, all necessary:
Offline evals (CI gate). A golden set of 50-100+ realistic tasks with expected behaviors, run on every prompt, model, or retrieval change. Scored by exact/fuzzy match where possible, rubric-based LLM-as-judge where necessary, human review for the cases judges disagree on. This is your regression gate; no prompt or model change ships without passing it.
Online evals (production sampling). Continuously score sampled live traffic with an automatic judge; humans review a sample of the sample. This is the only eval tier that sees real distribution shift and vendor model changes.
Task-level metrics (always on). Independent of judge quality, instrument coarse outcome signals: thumbs up/down, escalation-to-human rate, retry rate, tool-call success rate, answer-cited-source rate. These are cheap, robust signals that corroborate or contradict your eval scores.
On LLM-as-judge, honestly: judges are useful and imperfect. They exhibit position bias, verbosity bias, and self-preference (a model grades its own family generously). Mitigations that work in practice: pairwise comparison over absolute scoring where feasible, randomized position order, a cheap judge for routine scoring with escalation to stronger judges or humans for disputed cases, and periodic calibration of judge verdicts against a human-labeled set. Never let one judge's uncalibrated score be your only quality signal.
Layer 3: Drift — detecting the changes you didn't ship
Drift in LLM systems has more sources than in classical ML, and the taxonomy matters because each type has a different detection method:
| Drift type | Cause | Detection signal |
|---|---|---|
| Input drift | Users ask new things; traffic mix shifts | Embedding distribution of inputs; topic mix change |
| Data/retrieval drift | Corpus changes, index degrades | Retrieval score distributions; freshness lag metrics |
| Model drift | Vendor updates/deprecations your model | Version pinning alerts; eval score drops on fixed golden set |
| Quality drift | Prompt changes, interaction effects | Online eval scores; task metrics week-over-week |
Alerting on quality is statistical. LLM outputs are noisy; a 2-point dip in a daily eval score of 40 samples is often noise, and a persistent 3-day trend of 1-point dips is often real. Use control-chart thinking (rolling baselines, minimum sample sizes, trend tests) rather than static thresholds, or your quality alerts will be ignored within a month.
The reference SLO set
Every production LLM system we ship carries at least these SLOs, reviewed weekly:
- Task success rate — the quality SLO; measured by online evals and outcome metrics
- p50/p95 task latency — end-to-end, per user task, not per model call
- Cost per successful task — divides spend by successes, exposing "cheap but useless" failures
- Tool-call failure rate — agents die through their tools more often than through the model
- Escalation rate — how often the system hands off to humans
Cost-per-successful-task deserves special mention: it is the single metric that resists both cost-cutting that destroys quality and quality theater that destroys margins.
Privacy: the constraint that shapes everything
Traces contain user content; evals multiply it. Three rules:
1. Redact at the edge. PII removal happens in the agent runtime before telemetry leaves the process — typed pipeline stage, tested like any other code, not a regex sprinkled in later. 2. Sample down, not everything. Most telemetry value survives aggressive sampling; most privacy risk does not survive not collecting it. 3. Local inference changes the calculus. Running models on-prem lets you keep full-fidelity traces inside your boundary — a genuine debugging advantage of private AI deployments that teams rarely price in when comparing inference options.
A pragmatic rollout order
If you are starting from zero, in order of return on effort:
1. OTel-instrumented traces with cost and token accounting (week one). 2. One task-level outcome metric per workflow (week one). 3. A 50-query golden set gating every change in CI (first month). 4. Sampled online evals with a cheap judge, humans on disputes (first quarter). 5. Drift monitors and statistical alerting (ongoing).
Teams that do steps 1-3 catch the majority of production quality incidents. Everything after that is speed and precision.
References and further reading
- OpenTelemetry GenAI semantic conventions (official specification)
- LangSmith, Braintrust, Langfuse, and Arize Phoenix documentation on tracing and evaluation
- Zheng, L. et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (arXiv:2306.05685)
- Chung, H. W. et al., "Scaling Instruction-Finetuned Language Models" — evaluation methodology appendix
- "ML Test Score" rubric (Sculley et al., NeurIPS 2019 tutorial on ML production readiness)
- Databricks, "OLTP for LLMs: Measuring Drift in RAG Applications" engineering blog
- Google SRE Book, chapters on SLIs/SLOs — the statistical alerting foundations still apply
- NIST AI RMF 1.0 — measurement and monitoring guidance
Want an observability retrofit for a production LLM system — traces, evals, and drift monitoring wired in properly? Contact us. We write about production AI engineering regularly on the blog.