Lesson 7 of 10 Patterns

Redis Pub/Sub vs Redis Streams

Redis Pub/Sub for live broadcast versus Redis Streams for durable messaging: consumer groups, XACK, pending entries and reclaiming with XAUTOCLAIM.

45-55 min intermediate 0/10 lessons done
Course contents Lesson 7 of 10 · 0 done
On this page (8 sections)
  1. What you will learn
  2. The code
  3. Pub/Sub
  4. Streams
  5. Lists, Pub/Sub, Streams or Kafka?
  6. Key takeaways
  7. Exercises
  8. Quiz

What you will learn

  • Pub/Sub: real-time fan-out with no storage.
  • Streams: an append-only log with IDs, consumer groups and acknowledgements.
  • Recovering messages from a crashed consumer.

The code

"""Pub/Sub vs Streams: fire-and-forget broadcast versus a durable log with consumer groups."""
import os
import threading
import time

import redis

r = redis.Redis(port=int(os.getenv("REDIS_PORT", 6379)), decode_responses=True)
r.flushdb()

# --- Pub/Sub: only subscribers connected right now receive the message ---------
received = []
sub = r.pubsub()
sub.subscribe("chat:room1")


def listen():
    for msg in sub.listen():
        if msg["type"] == "message":
            received.append(msg["data"])
            if len(received) == 2:
                return


t = threading.Thread(target=listen)
t.start()
time.sleep(0.1)
print("delivered to", r.publish("chat:room1", "hello"), "subscriber(s)")
r.publish("chat:room1", "anyone here?")
t.join()
sub.close()
print("subscriber got:", received)
print("publish with no subscribers reaches:", r.publish("chat:empty", "lost"))

# --- Streams: messages are stored; consumer groups share work and acknowledge it --
for i in range(1, 6):
    r.xadd("orders", {"order_id": i, "amount": 100 * i})
r.xgroup_create("orders", "billing", id="0")

batch_a = r.xreadgroup("billing", "worker-a", {"orders": ">"}, count=3)
batch_b = r.xreadgroup("billing", "worker-b", {"orders": ">"}, count=3)
ids_a = [mid for mid, _ in batch_a[0][1]]
ids_b = [mid for mid, _ in batch_b[0][1]]
print("worker-a got", len(ids_a), "| worker-b got", len(ids_b))

r.xack("orders", "billing", *ids_a)  # worker-a finished; worker-b crashes before acking
print("pending:", r.xpending("orders", "billing")["pending"])

time.sleep(0.2)
# worker-c claims messages that have been pending for more than 100 ms
claimed = r.xautoclaim("orders", "billing", "worker-c", min_idle_time=100, start_id="0-0")
reclaimed = [mid for mid, _ in claimed[1]]
r.xack("orders", "billing", *reclaimed)
print("worker-c reclaimed", len(reclaimed), "| pending now:", r.xpending("orders", "billing")["pending"])
print("stream length (history kept):", r.xlen("orders"))

Output:

delivered to 1 subscriber(s)
subscriber got: ['hello', 'anyone here?']
publish with no subscribers reaches: 0
worker-a got 3 | worker-b got 2
pending: 2
worker-c reclaimed 2 | pending now: 0
stream length (history kept): 5

Pub/Sub

PUBLISH channel message delivers to every client currently SUBSCRIBEd, and returns how many received it. There’s no storage: a subscriber that’s offline, or connects a moment later, misses the message (the final publish reached 0 subscribers and is gone). Pattern subscriptions (PSUBSCRIBE chat:*) and sharded pub/sub in Cluster are available.

Good for: live notifications to connected clients, cache-invalidation broadcasts, chat typing indicators. Not for anything that must be processed.

Streams

A stream is an append-only log. XADD appends an entry with an auto-generated ID like 1732212345678-0 (milliseconds-sequence), and entries stay until trimmed (MAXLEN or MINID).

Consumer groups share work, Kafka-style:

  • XGROUP CREATE orders billing 0 creates a group that starts from the beginning.
  • XREADGROUP ... > gives each consumer new entries not yet delivered to anyone in the group. worker-a got three, worker-b the remaining two.
  • Delivered entries sit in the group’s pending entries list (PEL) until the consumer XACKs them.
  • If a consumer dies, its pending entries can be taken over: XAUTOCLAIM transfers entries idle longer than a threshold to another consumer. worker-c reclaimed the two that worker-b never acknowledged.

This gives at-least-once delivery: a message may be processed twice (if a worker processed it but crashed before XACK), so handlers should be idempotent.

Lists, Pub/Sub, Streams or Kafka?

NeedChoose
Broadcast to whoever is online, loss OKPub/Sub
Simple job queue, small scaleList with BRPOP/LMOVE
Durable messages, consumer groups, replay, acksStreams
Very high volume, long retention, many independent readersKafka (see the Kafka course)

Streams are bounded by memory, so set a MAXLEN ~ N or MINID trimming policy.

Key takeaways

  • Pub/Sub is fire-and-forget: only currently connected subscribers receive messages.
  • Streams store entries; consumer groups split work and track unacknowledged entries.
  • XACK after processing; XAUTOCLAIM recovers messages from dead consumers.
  • Delivery is at-least-once, so make handlers idempotent, and trim streams to bound memory.

Exercises

  1. Add maxlen=1000, approximate=True to XADD and confirm the stream stays bounded.
  2. Make a worker that retries a message three times, then moves it to a orders:dead stream.
  3. Use XRANGE to replay all orders from a given ID, for example to rebuild a report.

Quiz

  1. What happens to a Pub/Sub message when nobody is subscribed?
  2. What is the pending entries list?
  3. Why should stream consumers be idempotent?
Show answers
  1. It’s dropped; PUBLISH returns 0 and nothing stores it.
  2. The per-group list of entries delivered to a consumer but not yet acknowledged with XACK.
  3. Delivery is at-least-once: an entry processed before a crash but not acknowledged will be delivered again.

Finished this lesson?

Mark it complete to track your progress (saved in this browser).