You Can't Debug What You Can't See: Observability, Monitoring & Logging in Microservices

It was 2:47 AM when the alert fired.

Orders were failing. Not all of them — about 23% — and only for users in a specific region. The on-call engineer had six services to look at, logs spread across three different systems, no correlation between them, and a growing queue of customer complaints piling up in Slack.

Four hours later, the root cause turned out to be a single misconfigured timeout in a payment gateway client. Four lines of code. But finding those four lines took four hours because nobody could answer the most basic question: which service is actually broken?

I've been in that room. I've also been in rooms where the same class of incident takes twelve minutes to diagnose, not four hours. The difference isn't the team's skill level. It's whether the system was built to be understood.

That's what observability means in practice. Not dashboards. Not log files. The ability to ask arbitrary questions about your system's internal state — in production, under pressure, at 2:47 AM — and get answers fast enough to matter.

Here's what I've learned about building systems that are actually observable.

Why Microservices Make This Hard

In a monolith, debugging is annoying but tractable. You have one process, one log file, one database. When something breaks, you grep the logs, look at the stack trace, find the problem.

Microservices shatter that. A single user request might touch eight services, three databases, two external APIs, and a message queue — all in under 200 milliseconds. When that request fails, the error surfaces at the edge (the API gateway returns a 500) but the cause is buried somewhere in the middle of that chain.

Without the right instrumentation, you're doing archaeology: digging through timestamps across disconnected log streams, trying to reconstruct what happened from fragments. With the right instrumentation, you pull up a single trace and see the entire request like a timeline — every service, every database call, every millisecond.

The Three Pillars — And Why You Need All of Them

Every observability conversation eventually comes back to three signals. They're not interchangeable. Each answers a different question.

Logs tell you what happened: the events, the errors, the context around a specific moment. They're your highest-resolution signal — every log entry is a data point — but they don't scale as an alert mechanism. You can't alert on a grep.

Metrics tell you how the system is behaving over time. They're aggregates: how many requests per second, what percentage are failing, what's the p99 latency. They're cheap to store, fast to query, and perfect for dashboards and alerts. But they tell you something is wrong, not why.

Traces tell you where the time went. A trace follows a single request across every service it touched, showing you the exact sequence of calls, the duration of each hop, and where latency or errors originated. They're the bridge between "the dashboard says something is broken" and "I know exactly which service and which call is broken."

You need all three. Metrics catch the problem. Traces locate it. Logs explain it.

Distributed Tracing: Following a Request Across Services

Tracing is the hardest of the three pillars to implement correctly, and the most valuable once you have it.

The core concept is straightforward. When a request enters your system, you generate a globally unique trace ID. Every service that handles that request — or spawns a child request to another service — attaches that trace ID to its own work. Each unit of work becomes a span. All the spans for one trace ID, assembled in order, give you the complete picture of what happened.

Without the trace, you see a 500 error at the API gateway. With the trace, you see that 12ms was spent in inventory (fine) and 3000ms timed out in payment (the problem). You know exactly where to look.

The industry standard for implementing this is OpenTelemetry — an open-source framework that provides instrumentation libraries for most languages and frameworks, and exports data in a vendor-neutral format. Instrument once, send to any backend (Jaeger, Tempo, Datadog, Honeycomb, whatever). Don't use proprietary SDKs that lock you into a vendor's tracing format.

Two things to get right from day one:

Propagate the trace ID everywhere. HTTP headers (traceparent), message queue message headers, async job metadata. If the trace ID doesn't cross a boundary, the trace breaks there.

Sample intelligently. You cannot store 100% of traces at production scale — it's too expensive. But you must keep 100% of error traces and 100% of slow traces. Head-based sampling (decide at the start of the request) is simpler; tail-based sampling (decide after you've seen the full trace) gives you better coverage of anomalies. Start with head-based at 10% + always-sample on errors.

Structured Logging: Stop Writing Sentences, Start Writing Data

Most logging looks like this:

[2026-08-14 02:47:13] ERROR: Payment failed for user 12345 after 3000ms

This is readable to humans. It's nearly useless to machines, and machines need to query it at 3 AM when you're asleep.

Structured logging looks like this:

{
  "timestamp": "2026-08-14T02:47:13.441Z",
  "level": "error",
  "service": "payment-service",
  "trace_id": "abc-123-def-456",
  "span_id": "xyz-789",
  "user_id": "12345",
  "order_id": "ord-98765",
  "event": "payment_failed",
  "reason": "gateway_timeout",
  "duration_ms": 3000,
  "gateway": "stripe",
  "environment": "production",
  "version": "2.4.1"
}

Every log entry is a JSON object with consistent fields. You can filter by service:payment-service AND level:error AND duration_ms:>2000. You can correlate with trace_id:abc-123 and see every log entry across every service for that one request. You can alert on error rates per service using log-based metrics.

The fields you must always include, without exception:

• trace_id and span_id — links the log entry to its distributed trace
• service — which service emitted this log
• environment — prod vs staging, no ambiguity
• version — which deployment, critical for release correlation
• user_id or session_id — for user-impacting issues
• duration_ms on any operation that has a duration

What not to log: PII (email addresses, card numbers, passwords — ever), request/response bodies by default (too large, often sensitive), debug-level noise in production. Set your production log level to warn or error on most services; info for genuinely useful lifecycle events.

Metrics and Alerting: The RED Method

For microservices, the RED method gives you the three metrics that matter most for every service:

• Rate — requests per second (is the service being used?)
• Errors — error rate as a percentage (is it working?)
• Duration — latency distribution, especially p99 (is it fast?)

These three metrics, applied consistently across every service, give you a baseline health view of your entire system. If a service's error rate spikes or its p99 latency doubles, you'll know within seconds.

Alert on symptoms, not causes. Alert on user-facing error rate going above 1%, not on CPU usage going above 70%. CPU at 70% might be fine. User errors at 1% are never fine. Cause-based alerts produce false positives and alert fatigue — engineers start ignoring pages because most of them turn out to be nothing.

Define SLOs before you define alerts. A Service Level Objective (SLO) is a specific reliability target: "99.5% of order requests succeed within 500ms, measured over a rolling 30-day window." Your alert is a burn rate alarm — firing when you're consuming your error budget fast enough to breach the SLO if unchecked. This approach produces far fewer alerts, all of them meaningful.

The fastest path to alert fatigue is alerting on infrastructure metrics with no clear user impact. The fastest path to reliable on-call is alerting on what users actually experience.

The Observability Stack

You don't need to spend six figures on observability tooling. The open-source stack built around OpenTelemetry is production-grade and runs well on modest infrastructure:

OpenTelemetry Collector sits between your services and your storage backends. It receives telemetry in the OpenTelemetry Protocol (OTLP), does filtering and sampling, and fans out to multiple backends. Centralising this logic in the collector means you can change backends without touching service code.

Prometheus for metrics. Pull-based, efficient, battle-tested at massive scale. Write your alert rules in PromQL, store them in version control alongside your service code.

Loki for logs. Unlike Elasticsearch, Loki doesn't index log content — it only indexes labels (service, environment, level). This makes it dramatically cheaper for high-volume logs. Query with LogQL, which is deliberately similar to PromQL so the mental model transfers.

Tempo for traces. Stores full traces cheaply using object storage (S3, GCS). Integrates with Grafana so you can jump from a metric spike → related traces → related logs without leaving the same UI.

Grafana as the single pane of glass across all three. The "Explore" view lets you correlate across Prometheus, Loki, and Tempo in the same interface. When your alert fires, you open one URL, not three different systems.

For teams that want managed infrastructure, Grafana Cloud has a generous free tier and the same OSS stack under the hood. For teams at larger scale, Datadog and Honeycomb both provide excellent managed observability — at a cost that is absolutely worth it if you have the budget, because good tooling pays for itself in reduced MTTR.

The Mistakes I See Most Often

No correlation IDs across async boundaries. Teams instrument HTTP calls correctly but forget message queues. A trace that started in the API stops at the queue boundary and never reaches the consumer. Fix: always propagate trace_id in message metadata, and read it in the consumer to continue the trace.

Logging at debug level in production. The volume is unmanageable and the cost is real. Debug logs exist for local development. Production log levels should be warn or error for most services. Use dynamic log level adjustment (most logging frameworks support it) if you need to temporarily increase verbosity for a specific service during an incident.

Alerting on everything. I've seen monitoring setups with 200 alert rules. When 200 things can page you, nothing feels urgent. Start with RED metrics per service plus a handful of SLO burn rate alerts. Add more only when you identify a real gap that caused an incident.

No baseline. You can't know if p99 latency of 450ms is good or bad without knowing what it normally is. Set up baseline dashboards in week one — before you have any incidents — so you know what "normal" looks like. Anomalies only become visible against a baseline.

Treating observability as an ops concern. If only the platform team can add instrumentation, you will always have gaps where something goes wrong and nobody can see inside it. Observability is a development practice. Every service team owns the instrumentation of their service, including adding the trace attributes and log fields that make their service debuggable by someone who has never seen the code.

What Good Looks Like

A well-instrumented microservices system has a specific feel when something goes wrong.

The alert fires. It fires on a symptom — error rate on the order API is above the burn rate threshold. You open Grafana, find the service on the RED dashboard. Error rate spiked at 02:47. You click "show traces" in the time window. You see 200 failed traces, all with a span failing in payment-service. You click one trace. The payment service span shows a 3000ms timeout to the Stripe gateway. You click "show logs" for that span. The log entry says reason: "gateway_timeout", gateway: "stripe", retry_count: 3.

Twelve minutes from alert to root cause. Not four hours.

That's what you're building toward. Every piece of instrumentation you add — every trace ID propagated across a queue boundary, every structured log field, every SLO burn rate alert — reduces the time between "something is wrong" and "I know exactly what and why."

The systems that are genuinely reliable at scale aren't the ones that never break. They're the ones where breaking is understood quickly, fixed confidently, and prevented systematically — because the people running them can actually see inside.

I write about building production systems at scale. If you're setting up observability for a microservices platform and want to talk through the architecture, reach out.