TOPIC 04 — VISIBILITY
Monitoring & Observability
You can't fix what you can't see. Monitoring answers “is it broken?” — observability answers “why?” Great SRE teams are ruthless about both.
Monitoring vs observability
Monitoring watches for failure modes you predicted: dashboards, thresholds, alerts. Observability is a property of the system — enough telemetry, with enough context, that you can debug failures you never predicted by asking new questions of data you already collected. You need both: monitoring for the known unknowns, observability for the unknown unknowns that cause your worst nights.
The three pillars
Metrics
Numeric time series: request rates, error counts, latency histograms. Cheap to store, fast to query, ideal for alerting and SLOs. (Prometheus, Datadog)
Logs
Timestamped event records with full detail. Expensive at scale but irreplaceable for forensics. Structure them (JSON) or regret it. (Loki, Elasticsearch)
Traces
One request's journey across every service it touched, with timing at each hop. The only way to find the slow link in a 40-microservice chain. (Jaeger, OpenTelemetry)
A metric fires the alert (“p99 latency doubled”), a trace shows where (“the payment service's DB call”), and the logs show why (“connection pool exhausted after the 14:02 deploy”). Alert → locate → explain.
The four golden signals
If you can measure only four things about a user-facing system, measure these:
| Signal | What it is | The subtlety people miss |
|---|---|---|
| Latency | How long requests take | Track successful and failed latency separately — fast errors poison your average. |
| Traffic | Demand on the system (req/s, sessions) | Context for everything else. A 0% error rate at zero traffic is not health, it's silence. |
| Errors | Rate of failed requests | Include the sneaky ones: HTTP 200s with wrong content, and “success” that took 30 seconds. |
| Saturation | How full the system is | Systems degrade before 100%. Alert on the trend (“disk full in 4 hours”), not the cliff edge. |
The alerting philosophy
SRE alerting has one commandment: every page must be urgent, actionable, and user-visible. Everything else is a ticket or a dashboard line. The classic sorting:
- Page — a symptom users feel now: SLO burn rate spiking, checkout failing. Wake a human.
- Ticket — needs a human this week, not this minute: disk trending full, cert expiring in 14 days.
- Dashboard / log — everything else. Useful during debugging, silent otherwise.
A team paged 20 times a night stops reading pages — and then misses the real one. Noisy alerts aren't a nuisance, they're a direct threat to your SLO. Review every page weekly: was it actionable? If not, fix the threshold or delete it.
This is why burn-rate alerts on SLOs beat threshold alerts on machines: “CPU > 90%” pages you about a computer working hard; “error budget burning 14× too fast” pages you about users having a bad day. Only one of those deserves 3am.
Common pitfalls
- Dashboards nobody looks at, measuring things nobody chose. Instrument for questions you'll actually ask.
- Alerting on every cause you can imagine. You'll drown — and still miss the failure you didn't imagine.
- Averages everywhere. p50 hides the suffering of the p99 user, who is often your biggest customer.
- Unstructured log lines.
printf("something broke")at 3am helps no one. - No runbook link in the alert. A page should arrive with its own instructions.