LLM Observability in Production: Logs, Traces, and Metrics That Matter

Standard application monitoring isn't enough for LLM apps. A 200 OK with a confident-sounding but hallucinated answer looks fine to your metrics stack. LLM observability means tracking the things that tell you whether your AI is actually working — latency distribution, token spend, error and refusal rates, retrieval quality — and wiring them so you can debug, alert, and improve. Here's how to do it.

Why LLM apps need different observability

In a traditional service, a successful response is a successful response. In an LLM app:

• A hallucination returns 200 with a plausible-sounding wrong answer.
• A refusal returns 200 but produces no useful output.
• Latency variance is much wider (1s to 30s+), so percentiles matter more.
• Cost is embedded in the response (usage.prompt_tokens, usage.completion_tokens) — you need to extract and aggregate it.
• Retrieved context in a RAG app directly determines answer quality — if retrieval degrades, so does the model.

How the signals flow

OpenTelemetry · Observability Signal Flow

The four signal classes to instrument

1. Latency

Track p50, p95, and p99 per endpoint and model tier. p99 is where user experience breaks.

• Total request latency (end to end)
• Time-to-first-token (for streaming responses — what users feel as "responsiveness")
• LLM call latency vs. retrieval latency vs. your own processing (understand where time goes)

Alert on p95 > your SLA threshold, and on any sudden spike.

2. Token usage and cost

Extract from every response and aggregate by feature, model, and tenant:

usage = response.usage
input_tokens  = usage.prompt_tokens
output_tokens = usage.completion_tokens
cost = (input_tokens * INPUT_PRICE) + (output_tokens * OUTPUT_PRICE)

Track: tokens/request (input vs output), cost/request, cumulative cost/day. Alert on unexpected spikes — a prompt change that doubles token usage hits your bill before you notice. See token budgeting for sizing.

3. Error and refusal rates

Hard errors (rate limit, timeout, invalid request) are easy — they throw. Soft errors are subtle:

• Refusals ("I can't help with that") — the model responded but produced no useful output.
• Truncation — response hit max_tokens and was cut off.
• JSON parse failures — structured-output requests that returned malformed JSON.

Detect these with a post-response check and log them as a separate metric. A rising refusal rate often signals a prompt that's drifting toward policy boundaries.

4. Retrieval quality (for RAG)

• Retrieval latency — vector search is usually fast (< 100ms), but embedding calls are not.
• Chunk count returned — are you consistently getting your target topK?
• Similarity scores — if your vector store returns them, log the top-1 score. A low score means the query hit your corpus but nothing was actually relevant.
• Citation accuracy (offline) — periodically sample answers and check whether cited documents actually support the answer.

A structured log pattern for LLM calls

Emit one structured log event per LLM call with everything you need to debug later:

{
  "event": "llm_call",
  "trace_id": "abc123",
  "span_id": "def456",
  "feature": "ask_article",
  "model": "llama-3.1-8b-instant",
  "latency_ms": 1842,
  "input_tokens": 3102,
  "output_tokens": 248,
  "cost_usd": 0.00047,
  "finish_reason": "stop",
  "retrieval_chunks": 4,
  "retrieval_top_score": 0.87,
  "error": null
}

Key principles:

• Never log the prompt or completion — they will contain user data and PII. Log metadata, not content.
• Always log finish_reason — stop (clean), length (truncated), content_filter (refused).
• Attach a trace_id so you can join LLM calls to the user request they belong to.

Tracing across services

For multi-step pipelines (retrieve → rerank → generate), use distributed tracing. Each step gets a span under the same trace ID:

trace_id: abc123
  span: retrieve        latency=45ms  chunks=4  top_score=0.87
  span: rerank          latency=12ms  chunks=3
  span: llm_generate    latency=1842ms  tokens=3350  cost=$0.00047

OpenTelemetry is the standard instrumentation layer. Langfuse is purpose-built for LLM tracing and integrates directly with most frameworks (including Spring AI).

Alerting thresholds to start with

Start conservative — too many alerts train people to ignore them.

Tool landscape

• Langfuse — purpose-built LLM observability; traces, evals, cost tracking, prompt versioning. Free tier is generous; self-hostable.
• OpenTelemetry — vendor-neutral instrumentation for spans and metrics; exports to your existing stack (Prometheus, Jaeger, Grafana).
• Prometheus + Grafana — standard metrics/dashboards; add your LLM metrics via custom counters.
• AI gateway (Apigee, Kong AI, LiteLLM) — centralized token metering and cost allocation without instrumenting every service.

FAQ

Do I need all of this from day one? No. Start with structured logging (the JSON pattern above) and token cost tracking — those two give you the most signal for the least work. Add tracing and eval when you're debugging retrieval or multi-step failures.

How do I avoid logging PII? Log metadata (token counts, scores, latency, IDs) never prompt content or completions. For compliance, make this a code standard, not a documentation note.

What's the difference between monitoring and evaluation? Monitoring is real-time signal (is it up, is it fast, is it spending what it should?). Evaluation is offline quality assessment (are the answers good?). You need both — see testing AI applications.

My LLM app is simple — do I still need this? At minimum: log token usage per request and track the error rate. Everything else scales with complexity.

LLM observability is the difference between flying blind and knowing when your AI degrades. Instrument latency, tokens, errors, and retrieval quality; emit structured logs without PII; trace multi-step pipelines; and alert on the signals that actually predict user experience.

More: the LLM Observability Guide, AI gateway for centralized metering, and token budgeting.

Shipping an LLM app to production and want help with your observability stack? Let's talk.