Observability for honest systems

A system should explain itself

When production fails, the worst answer is silence. The second worst is a wall of green dashboards while users are visibly hurting. I have debugged both, and they share a root cause: telemetry that was added to satisfy a checklist rather than to answer questions.

The distinction I draw between monitoring and observability is practical, not academic. Monitoring answers the questions you predicted: is the CPU high, is the disk full, is the endpoint up. Observability lets you ask questions you did not predict: why is checkout slow for exactly one tenant, only on retries, only since Tuesday's deploy. Production incidents live almost entirely in the second category.

An honest system is one that can testify about its own behavior — including the behavior you did not expect it to have.

Metrics, logs, traces — what each is for

The three signal types are complements, not competitors, and most telemetry budgets are wasted by using one where another belongs:

• Metrics are cheap, aggregated, and fast to query. They tell you that something is wrong and how much. They are the right backbone for alerting because they are stable and inexpensive at scale.
• Traces follow a single request across services. They tell you where the time or the failure went. One good trace often replaces an hour of log spelunking across five services.
• Logs carry arbitrary detail. They tell you why — the payload shape, the branch taken, the exception context. Structured, correlated logs are gold; unstructured ones are a write-only archive.

The glue is correlation: a trace identifier stamped on every log line and exemplars linking metrics to traces. Without correlation you own three separate archives; with it you own one investigative surface where each signal hands off to the next.

Signals before dashboards

The common failure mode is dashboard-first observability: build the Grafana board, fill it with every metric the runtime exports, call it done. Those boards look impressive and answer nothing, because nobody decided what question each panel exists to answer.

I start from the opposite end — the user-impact signals, each tied to a question and an owner:

Queue age deserves the special mention it gets in that table. Queue depth looks reassuring while a poison message blocks the head of the line; age exposes the problem immediately. Measure how long the oldest item has waited, not how many items are waiting.

Dashboards then become a view over these signals — one service, one screen, impact at the top, causes below. A dashboard that requires scrolling during an incident is a dashboard that will be abandoned during an incident.

Trace the boundary

You cannot instrument everything, and you should not try. Tracing earns its cost at boundaries: API calls, message handlers, database queries, background jobs, third-party integrations. That is where latency hides, where retries multiply, and where responsibility changes hands — which makes boundaries exactly where the arguments happen during an incident.

Attach the attributes you will actually filter by. In multi-tenant systems the tenant identifier is non-negotiable; per-tenant latency breakdown has located more production mysteries for me than any other single query.

activity?.SetTag("tenant.id", tenantId);
activity?.SetTag("operation.name", "invoice.calculate");
activity?.SetTag("queue.age_ms", queueAge.TotalMilliseconds);
activity?.SetTag("retry.attempt", attempt);
activity?.SetTag("peer.service", "payments-gateway");

Use OpenTelemetry and its semantic conventions rather than inventing attribute names. The payoff is not ideological — it is that every backend, dashboard template, and future teammate already understands http.response.status_code, and none of them understands myStatusCode2.

Cardinality is the real bill

Telemetry pricing has a simple physics: metrics cost per unique label combination, logs and traces cost per byte. The mistake that produces shock invoices is putting unbounded values — user IDs, URLs with parameters, session tokens — into metric labels. One careless label on a busy counter can mint millions of time series.

For traces at volume, sample — but sample with intent. Head sampling at a few percent keeps costs flat and is fine for latency analysis; tail sampling that keeps every error and every slow request preserves precisely the traces you will want at 3 a.m. Most teams I work with land on a hybrid: low baseline head sampling plus tail rules for errors and outliers.

Alerts people trust

Alert fatigue is not a tooling problem; it is a design debt that compounds. Every alert that fires without requiring action teaches the on-call engineer to ignore alerts, and that lesson transfers to the one alert that mattered.

The standard I hold alerts to: an alert page means a human decision is needed now. Everything else is a ticket or a dashboard annotation. In practice:

• Alert on symptoms (user-facing latency, error rate, budget burn), not on causes (CPU, memory, pod restarts). Causes belong on the dashboard you open after the symptom fires.
• Every alert links to a runbook — even three lines: what this means, what to check first, who to escalate to.
• Every fired alert gets reviewed weekly: was it actionable? If not, it gets retuned or deleted. No exceptions, or the set only ever grows.
• Burn-rate alerting over static thresholds for SLOs — it pages fast for fast burns and stays quiet for slow drifts that a ticket can handle.

Production considerations

• Instrument before you need it. The trace you wish you had during an incident cannot be added retroactively. Boundary instrumentation goes in with the feature, as part of the definition of done.
• Test your telemetry. A wrong metric is worse than no metric — it lends false confidence. Assert in integration tests that key events and counters actually emit with the expected attributes.
• Watch the collector like a service. The observability pipeline itself fails: dropped spans, lagging exporters, full buffers. Monitor its throughput, or your first sign of trouble is a suspiciously quiet dashboard.
• Retention is a decision, not a default. Thirty days of traces, ninety of metrics, and an archive tier for compliance logs covers most products. Decide deliberately — vendor defaults optimize for the vendor.

Summary

• Observability means answering unpredicted questions; monitoring only answers predicted ones. Incidents live in the first category.
• Metrics tell you that, traces tell you where, logs tell you why — correlation makes them one surface.
• Define user-impact signals with owners before building any dashboard.
• Instrument boundaries first; stamp tenant and operation attributes you will filter by.
• Keep metric labels bounded; put unbounded data in traces and logs; sample with tail rules for errors.
• A page means a human decision is needed now — review every fired alert weekly and delete the noise.