Module 13Term 4 · Production operations8 poster items

Disaster Recovery and Incident Management

Sign in to save your progress across devices. Until then it is kept in this browser.

Why this matters

Every system eventually has a bad day: a provider outage, a bad migration, a deleted database, a breached key. Enterprises and regulators do not ask whether you will have incidents. They ask how fast you detect, how you communicate, how much data you can lose, and how fast you recover.

Learning outcomes

  1. Define RPO and RTO per system and design backups to meet them.
  2. Prove backups by restoring them.
  3. Plan failover and understand when multi-region is justified.
  4. Run an incident with clear roles and communication.
  5. Set up sustainable on-call.
  6. Write blameless postmortems that produce real fixes.
  7. Run controlled chaos experiments.

Lessons

13.1 RPO and RTO (83)

  • RPO (Recovery Point Objective): how much data you can afford to lose (e.g. 5 minutes).
  • RTO (Recovery Time Objective): how long you can be down (e.g. 1 hour).
  • Set them per system with the business. A ledger needs a near-zero RPO; an analytics cache can be rebuilt.

13.2 Backups (84)

  • 3-2-1 rule: 3 copies, 2 different media/services, 1 off-site (different account/provider).
  • Point-in-time recovery (PITR) for the primary database.
  • Backups immutable and in a separate account so a compromised admin or rogue agent cannot delete them. (Agents and attackers deleting backups alongside production is a recurring incident pattern.)
  • Back up more than the DB: object storage, secrets/config (via IaC), third-party data you cannot regenerate.
  • A backup you have not restored is a hope. Schedule restore drills; measure actual RTO.

13.3 Failover and multi-region (85, 86)

  • Failover (85): promote a standby (replica, secondary region, second provider). Automatic failover needs reliable health detection to avoid flapping or split-brain.
  • Multi-region (86): improves latency for global users and survives regional outages, at the cost of complexity, data-consistency challenges and money. Most startups need multi-zone within one region, plus off-region backups, not active-active multi-region.
  • Consider data residency before replicating personal data across borders.
  • Single-provider dependencies (one LLM vendor, one SMS gateway): plan a secondary.

13.4 Incident response (109)

Severity levels (example): SEV1 outage or data breach / money movement wrong; SEV2 major feature down; SEV3 degraded, workaround exists. Roles: Incident Commander (coordinates, decides), Ops/Tech lead (fixes), Comms lead (status page, customers, regulators), Scribe (timeline). In a one-person company you play all four; write the checklist in advance so you do not have to think. Sequence: detect → declare → mitigate first (rollback, flag off, failover) → communicate early and regularly → resolve → follow up. For breaches, legal clocks start: e.g. Singapore PDPA requires notifying the PDPC within 3 calendar days of assessing a breach as notifiable (see 14 · Compliance, Governance and Audit).

13.5 On-call (110)

Clear rotation, escalation path, runbooks for every alert, alert quality reviews, compensation/rest after pages. Solo founder version: a paging tool on your phone, a trusted contractor as backup, and ruthless alert hygiene.

13.6 Postmortems (111)

Blameless: focus on how the system allowed the failure, not who erred. Template: summary, impact (users, duration, money), timeline, root cause and contributing factors (5 whys), what went well, what went badly, where we got lucky, action items with owners and due dates. Track action items to completion.

13.7 Chaos engineering (87)

Form a hypothesis ("if the LLM provider fails, users see the fallback and nothing else breaks"), limit blast radius, run in staging first, then carefully in production during game days, measure, fix. Start with tabletop exercises before injecting real faults.

Lab 13: Game day

  1. Write RPO/RTO for each data store; configure PITR and off-account immutable backups.
  2. Restore drill: restore production backup into a fresh environment; verify data; record actual RTO.
  3. Game day: instructor breaks the system (bad deploy, leaked key, provider outage, deleted table) without warning. Students run the incident with roles, status page and comms.
  4. Write the postmortem within 48 hours.

Definition of Done

AI prompt pack

  • "Write an incident runbook for 'primary database unavailable' for this architecture, including detection, mitigation, comms and recovery steps."
  • "Turn this incident timeline into a blameless postmortem with 5-whys and concrete action items."