Lesson 8 of 15 Communication

Message Queues and Async Processing

Decouple services with message queues and event streams: queue vs log, delivery guarantees, idempotent consumers, retries, dead letters and backpressure.

35-45 min intermediate 0/15 lessons done
Course contents Lesson 8 of 15 · 0 done
On this page (10 sections)
  1. What you will learn
  2. Why put a queue in the middle
  3. Queue vs log
  4. Delivery guarantees
  5. Retries and dead letters
  6. Ordering
  7. Backpressure
  8. Key takeaways
  9. Exercises
  10. Quiz

What you will learn

  • What problems queues solve and when to use them.
  • The difference between a work queue and an event log.
  • At-most-once, at-least-once and “exactly-once” delivery.
  • Idempotent consumers, retries, dead-letter queues and backpressure.

Why put a queue in the middle

Suppose uploading a video must also transcode it, generate thumbnails, notify followers and update search. Doing all that inside the HTTP request makes it slow and fragile: if the search service is down, the upload fails.

With a queue, the upload service records the video and publishes an event VideoUploaded. Workers for each task consume at their own pace.

upload API --> [ queue / topic ] --> transcoder workers
                                 --> thumbnail workers
                                 --> notification service

Benefits:

  • Decoupling: producers don’t know or wait for consumers.
  • Load levelling: a spike fills the queue; workers drain it steadily instead of falling over.
  • Resilience: if a consumer is down, messages wait.
  • Scaling: add workers to drain faster.

Cost: the system becomes eventually consistent (the thumbnail appears a few seconds later), and you now operate another critical component.

Queue vs log

Work queue (RabbitMQ, SQS)Event log (Kafka, Pulsar, Kinesis)
Message lifetimeDeleted once acknowledgedRetained for days; consumers track an offset
ConsumersCompete: each message goes to one workerEach consumer group gets every message
ReplayNoYes, rewind the offset
OrderingPer queue, weakened by parallel workersPer partition
Typical useBackground jobs, tasksEvent streams, analytics, many subscribers

Our Kafka course covers logs in depth.

Delivery guarantees

  • At-most-once: acknowledge before processing. A crash loses the message. Fine for metrics you can afford to drop.
  • At-least-once: acknowledge after processing. A crash after processing but before the ack means the message is delivered again. The usual default.
  • Exactly-once: really means “effectively once”: at-least-once delivery plus deduplication or transactional processing.

Because duplicates happen, make consumers idempotent: processing the same message twice has the same effect as once. Techniques: store processed message ids and skip repeats; use upserts keyed by a natural id; make operations set values instead of incrementing them; include an idempotency key the downstream service enforces.

Retries and dead letters

Transient failures (a timeout) should be retried with exponential backoff and jitter (wait 1 s, 2 s, 4 s… plus randomness so retries don’t synchronise). Messages that keep failing (a malformed payload) go to a dead-letter queue after N attempts, so they stop blocking the line and a human or tool can inspect them.

Ordering

Strict global ordering kills parallelism. Usually you need order per entity: all events for one order, or one chat conversation, in sequence. Use the entity id as the partition key (Kafka) or message group id (SQS FIFO), so one entity’s events go to one partition and one consumer.

Backpressure

If producers outrun consumers forever, the queue grows without bound. Watch queue depth / consumer lag, autoscale consumers on it, and if needed slow producers down (reject or rate-limit, see lesson 11).

Key takeaways

  • Queues decouple services, absorb spikes and let slow work happen asynchronously.
  • Work queues hand each message to one worker; logs retain events for many consumer groups and replay.
  • Assume at-least-once delivery and make consumers idempotent.
  • Retry with backoff and jitter, park poison messages in a dead-letter queue, partition by entity for ordering, and monitor lag.

Exercises

  1. Write a Python consumer that processes (message_id, account, amount) events idempotently using a set of processed ids. Feed it a list with duplicates.
  2. Design the async pipeline for an e-commerce checkout: which steps stay synchronous, which become events?
  3. Compute the total wait before the 5th retry with exponential backoff starting at 0.5 s and doubling. Add ±20% jitter and explain why it helps.

Quiz

  1. Why do at-least-once systems need idempotent consumers?
  2. How do you keep all events for one order in sequence while still processing in parallel?
  3. What is a dead-letter queue for?
Show answers
  1. A message can be delivered more than once (e.g. crash after processing but before acknowledging), so processing must be safe to repeat.
  2. Partition by order id so each order’s events land in one partition consumed by one worker; different orders go to different partitions in parallel.
  3. Holding messages that repeatedly fail, so they don’t block other messages and can be inspected and fixed.

Finished this lesson?

Mark it complete to track your progress (saved in this browser).