Module 12Term 4 · Production operations11 poster items

Observability and SRE

Sign in to save your progress across devices. Until then it is kept in this browser.

Why this matters

If customers tell you it is broken before your alerts do, you do not have monitoring, you have customers doing QA. Observability is also what makes AI-written code debuggable: when you did not write the code, you need the system to tell you what it is doing.

Learning outcomes

  1. Emit structured logs, metrics and traces with OpenTelemetry.
  2. Define SLIs and SLOs that reflect user happiness, and use error budgets.
  3. Write alerts that are actionable and do not cause fatigue.
  4. Build dashboards for the four golden signals.
  5. Observe LLM and agent behaviour, cost and quality.

Lessons

12.1 The three signals (58 to 61, 66)

  • Logs (59): structured JSON, one event per line, with request_id, user_id/tenant_id (not PII), level, event, duration. Never log secrets, tokens, passwords, full card numbers or raw personal data. Set retention.
  • Metrics (61): cheap numeric time series. RED for services (Rate, Errors, Duration); USE for resources (Utilisation, Saturation, Errors).
  • Traces (60): one request followed across services, queues and LLM calls, with spans and timings. Propagate trace context through queues and webhooks.
  • Monitoring (58) watches known failure modes. Observability (66) lets you ask new questions of the running system without shipping new code. You need both.
  • Use OpenTelemetry so you are not locked into a vendor.

12.2 The four golden signals

Latency, traffic, errors, saturation (from Google's SRE practice). One dashboard per service, top of the page.

12.3 SLIs, SLOs and error budgets (63, 64, 65)

  • SLI (64): a measured ratio of good events: "requests served successfully in under 500 ms / total requests."
  • SLO (63): a target for the SLI over a window: "99.9% over 28 days."
  • Error budget (65): the allowed failure: 0.1% of 28 days is roughly 40 minutes. Budget left: ship freely. Budget burned: slow down and fix reliability.
  • Pick SLOs per user journey (login, checkout, AI answer), not per server. 100% is the wrong target: it is unaffordable and stops all change.

12.4 Alerts (62)

  • Page a human only when users are hurting now or soon (symptom-based), not on every CPU spike.
  • Prefer burn-rate alerts on SLOs (fast burn pages, slow burn tickets).
  • Every alert links to a runbook: what it means, how to check, how to mitigate.
  • Business alerts too: zero sign-ups in an hour, payment success rate drop, LLM spend spike, DLQ growing, cron heartbeat missed.
  • Review alert noise monthly; delete alerts nobody acts on.

12.5 Health checks and probes (56, 57)

See 07 · Resilience Patterns. Add external synthetic monitoring of the real journey from multiple regions.

12.6 LLM and agent observability

Trace every model call: model, version, prompt version, tokens in/out, cost, latency (time to first token and total), tool calls, retrieval sources, user feedback and eval scores. Tools: Langfuse, Helicone, Braintrust, or OpenTelemetry GenAI conventions into your existing backend. Watch for quality drift after model or prompt changes.

12.7 Error tracking and RUM

Exception tracking with source maps and release tags (Sentry or equivalent). Real-user monitoring of Core Web Vitals.

Lab 12: Know before they do

  1. Instrument the app with OpenTelemetry (logs, metrics, traces) end to end, including a queue hop and an LLM call.
  2. Define 3 SLOs for critical journeys; build the dashboard and burn-rate alerts.
  3. Write a runbook for each alert.
  4. Break something on staging; measure time from break to alert.
  5. Build an LLM cost and quality panel.

Definition of Done

AI prompt pack

  • "Add OpenTelemetry instrumentation to this service. Propagate context through the queue. Add a span around every LLM and tool call with token and cost attributes."
  • "Audit our logging calls and flag any that could log secrets, tokens or personal data."