Module 07Term 3 · Reliability and scale8 poster items

Resilience Patterns

Sign in to save your progress across devices. Until then it is kept in this browser.

Why this matters

Every external call (database, payment gateway, LLM provider, SMS) will at some point be slow, fail, or fail halfway. AI-generated code almost always assumes the happy path. A single slow dependency without a timeout can take your whole app down by exhausting threads, connections or serverless concurrency.

Learning outcomes

  1. Put a timeout on every outbound call and set it from a latency budget.
  2. Retry safely: only idempotent operations, with exponential backoff and jitter, with a cap.
  3. Use circuit breakers and fallbacks for graceful degradation.
  4. Apply backpressure and dead letter queues so overload and poison messages do not cascade.
  5. Expose meaningful health checks.

Lessons

7.1 Timeouts (11)

  • Default timeouts in many HTTP clients are infinite or minutes long. Set them explicitly: connect timeout + request timeout.
  • Derive from a budget: if the page must respond in 2 s and calls three services in sequence, each gets well under 2 s.
  • Propagate deadlines: if the caller has 300 ms left, do not start a 2 s call.
  • LLM calls: use streaming with first-token timeout and total timeout.

7.2 Retries and exponential backoff (12, 13)

  • Retry only transient errors (timeouts, 429, 502, 503, 504, connection resets). Never retry 400/401/403/422.
  • Retry only idempotent operations, or ones carrying an idempotency key (04 · APIs and Contracts).
  • Backoff: delay = min(cap, base * 2^attempt) plus random jitter so thousands of clients do not retry in sync (thundering herd).
  • Cap attempts (often 3) and total time. Honour Retry-After.
  • Retry at one layer only. Retries at every layer multiply: 3 x 3 x 3 = 27 calls per user action.

7.3 Circuit breakers (10)

States: closed (normal) → open after failure threshold (fail fast, no calls) → half-open after cool-down (let a few through to test) → closed if healthy. Pair with a fallback: cached data, a degraded feature, a queued "we'll notify you," or a secondary provider (e.g. second LLM vendor).

7.4 Bulkheads and backpressure (44)

  • Bulkheads: separate pools/limits per dependency so a slow LLM does not consume the capacity your login endpoint needs.
  • Backpressure: when consumers cannot keep up, push back (bounded queues, 429/503 with Retry-After, load shedding of low-priority work) instead of buffering until memory runs out.

7.5 Dead letter queues (20)

After N failed processing attempts, move the message to a DLQ instead of retrying forever (a "poison message" can block a queue). Alert on DLQ depth. Build a tool to inspect, fix and replay.

7.6 Health checks (56, 57)

  • Liveness: "is the process alive?" Cheap; never checks dependencies (otherwise a DB blip restarts every pod).
  • Readiness: "can I serve traffic now?" Checks critical dependencies; removes the instance from the load balancer when false.
  • Startup: for slow-booting services.
  • A public status page fed by synthetic checks of the real user journey.

7.7 Graceful degradation

Decide per feature: what does the user see when this dependency is down? Write it down. Examples: recommendations hidden, payments queued with clear messaging, AI assistant shows "temporarily unavailable" rather than a spinner forever.

Lab 7: Chaos on your laptop

Use Toxiproxy (or a fault-injection middleware) between your app and its dependencies.

  1. Add 5 s latency to the LLM provider. Observe. Add timeouts, retries with jitter, circuit breaker, fallback. Observe again.
  2. Make the payment provider return 503 for 50% of calls. Prove no double charges.
  3. Inject a poison message into your queue; prove it lands in the DLQ and alerts.
  4. Kill the database for 30 s. Readiness fails, liveness stays up, service recovers without restart.

Definition of Done

AI prompt pack

  • "Find every outbound network call in this repo. Report which lack a timeout, which retry non-idempotent operations, and which have no fallback."
  • "Create a shared HTTP client with timeouts, retries only on transient errors with exponential backoff and full jitter, and a circuit breaker per host."