Module 08Term 3 · Reliability and scale10 poster items

Async, Events and Real-Time

Sign in to save your progress across devices. Until then it is kept in this browser.

Why this matters

Sending email, generating a PDF, calling an LLM agent, processing a webhook: if these run inside the user's request, the request is slow, times out on serverless limits, and loses work on any crash. Async design is how serious systems stay fast and lose nothing.

Learning outcomes

  1. Move slow or unreliable work to durable background jobs.
  2. Choose between queues, pub/sub and event-driven designs.
  3. Handle webhooks safely (verify, dedupe, ack fast, process async).
  4. Coordinate multi-step workflows with sagas and the outbox pattern.
  5. Pick the right real-time transport: WebSockets, SSE or long polling.
  6. Run scheduled jobs that are safe to overlap and safe to miss.

Lessons

8.1 Queues and workers (15)

Pattern: request validates, writes intent to DB, enqueues job, returns 202 Accepted with a job ID. Worker processes, updates status, notifies.

  • Delivery is at-least-once in practice: design every job handler to be idempotent.
  • Options: managed (SQS, Cloud Tasks), Redis-based (BullMQ), Postgres-based (pg-boss, Graphile Worker), durable workflow engines (Inngest, Trigger.dev, Temporal).
  • Durable workflow engines are often the best fit for AI agents: each step is checkpointed and retried independently.

8.2 Pub/Sub and event-driven architecture (16, 17)

  • Queue: one message, one consumer (a task to do).
  • Pub/Sub: one event, many independent subscribers (a fact that happened: order.paid).
  • Event-driven: services react to events rather than calling each other directly. Decoupled, but harder to trace; requires schemas for events and good observability.
  • Name events in the past tense; version event schemas; include an event ID for dedupe.

8.3 Distributed transactions, outbox and sagas (18, 19)

  • You cannot have one ACID transaction across your DB, Stripe and an email provider.
  • Transactional outbox: in the same DB transaction, write the business change and an outbox row; a relay publishes outbox rows. Prevents "saved but never published" or "published but not saved."
  • Saga: a sequence of local steps each with a compensating action (reserve stock → charge card → create shipment; if shipment fails: refund, release stock). Orchestrated (a coordinator) or choreographed (events).
  • Two-phase commit exists but is rarely appropriate in modern web systems.

8.4 Webhooks (101)

Receiving:

  1. Verify the signature (HMAC with shared secret) and timestamp to stop forgery and replay.
  2. Dedupe on the provider's event ID (unique constraint).
  3. Ack fast (2xx within seconds), then process in a background job.
  4. Handle out-of-order delivery; fetch current state from the provider when in doubt. Sending: sign payloads, retry with backoff, provide a delivery log and replay for customers, guard against SSRF in customer-supplied URLs.

8.5 Cron jobs (21)

  • Assume a run can be skipped, delayed or run twice. Make jobs idempotent and resumable.
  • Prevent overlap with a lock (DB advisory lock).
  • Process in batches with checkpoints, not one giant transaction.
  • Monitor with a "dead man's switch" heartbeat (alert if the job did not run).
  • Store schedules in UTC; beware daylight saving in other markets.

8.6 Real-time transports (22, 23, 24)

Transport Direction Use when
SSE Server to client Streaming LLM tokens, notifications, progress. Simple, works over HTTP, auto-reconnect
WebSockets Both ways Chat, collaboration, live games, voice
Long polling Simulated push Fallback where streaming is blocked
Considerations: authenticate the connection, re-authorise on long-lived connections, handle reconnect and missed messages (resume from last event ID), serverless platforms may cap connection duration; use a managed real-time service (Supabase Realtime, Pusher, Ably, Cloudflare Durable Objects) at scale.

Lab 8: Make it async

  1. Move your slowest synchronous operation (LLM call, PDF, email) to a durable job with status polling or SSE progress.
  2. Implement a webhook receiver (Stripe or Resend) with signature verification, dedupe and async processing. Replay the same event 10 times; prove one effect.
  3. Implement a 3-step saga with compensation; inject failure at step 3; prove rollback.
  4. Add a cron job with overlap lock and heartbeat alert.

Definition of Done

AI prompt pack

  • "Identify every operation in request handlers that calls an external service or could take over 1 second. Propose which to move to background jobs."
  • "Implement this webhook handler: verify signature with constant-time compare, reject stale timestamps, dedupe on event id with a unique constraint, enqueue processing, return 200."