What you will learn
- Pub/Sub: real-time fan-out with no storage.
- Streams: an append-only log with IDs, consumer groups and acknowledgements.
- Recovering messages from a crashed consumer.
The code
"""Pub/Sub vs Streams: fire-and-forget broadcast versus a durable log with consumer groups."""
import os
import threading
import time
import redis
r = redis.Redis(port=int(os.getenv("REDIS_PORT", 6379)), decode_responses=True)
r.flushdb()
# --- Pub/Sub: only subscribers connected right now receive the message ---------
received = []
sub = r.pubsub()
sub.subscribe("chat:room1")
def listen():
for msg in sub.listen():
if msg["type"] == "message":
received.append(msg["data"])
if len(received) == 2:
return
t = threading.Thread(target=listen)
t.start()
time.sleep(0.1)
print("delivered to", r.publish("chat:room1", "hello"), "subscriber(s)")
r.publish("chat:room1", "anyone here?")
t.join()
sub.close()
print("subscriber got:", received)
print("publish with no subscribers reaches:", r.publish("chat:empty", "lost"))
# --- Streams: messages are stored; consumer groups share work and acknowledge it --
for i in range(1, 6):
r.xadd("orders", {"order_id": i, "amount": 100 * i})
r.xgroup_create("orders", "billing", id="0")
batch_a = r.xreadgroup("billing", "worker-a", {"orders": ">"}, count=3)
batch_b = r.xreadgroup("billing", "worker-b", {"orders": ">"}, count=3)
ids_a = [mid for mid, _ in batch_a[0][1]]
ids_b = [mid for mid, _ in batch_b[0][1]]
print("worker-a got", len(ids_a), "| worker-b got", len(ids_b))
r.xack("orders", "billing", *ids_a) # worker-a finished; worker-b crashes before acking
print("pending:", r.xpending("orders", "billing")["pending"])
time.sleep(0.2)
# worker-c claims messages that have been pending for more than 100 ms
claimed = r.xautoclaim("orders", "billing", "worker-c", min_idle_time=100, start_id="0-0")
reclaimed = [mid for mid, _ in claimed[1]]
r.xack("orders", "billing", *reclaimed)
print("worker-c reclaimed", len(reclaimed), "| pending now:", r.xpending("orders", "billing")["pending"])
print("stream length (history kept):", r.xlen("orders"))
Output:
delivered to 1 subscriber(s)
subscriber got: ['hello', 'anyone here?']
publish with no subscribers reaches: 0
worker-a got 3 | worker-b got 2
pending: 2
worker-c reclaimed 2 | pending now: 0
stream length (history kept): 5
Pub/Sub
PUBLISH channel message delivers to every client currently SUBSCRIBEd, and returns how many received it. There’s no storage: a subscriber that’s offline, or connects a moment later, misses the message (the final publish reached 0 subscribers and is gone). Pattern subscriptions (PSUBSCRIBE chat:*) and sharded pub/sub in Cluster are available.
Good for: live notifications to connected clients, cache-invalidation broadcasts, chat typing indicators. Not for anything that must be processed.
Streams
A stream is an append-only log. XADD appends an entry with an auto-generated ID like 1732212345678-0 (milliseconds-sequence), and entries stay until trimmed (MAXLEN or MINID).
Consumer groups share work, Kafka-style:
XGROUP CREATE orders billing 0creates a group that starts from the beginning.XREADGROUP ... >gives each consumer new entries not yet delivered to anyone in the group. worker-a got three, worker-b the remaining two.- Delivered entries sit in the group’s pending entries list (PEL) until the consumer
XACKs them. - If a consumer dies, its pending entries can be taken over:
XAUTOCLAIMtransfers entries idle longer than a threshold to another consumer. worker-c reclaimed the two that worker-b never acknowledged.
This gives at-least-once delivery: a message may be processed twice (if a worker processed it but crashed before XACK), so handlers should be idempotent.
Lists, Pub/Sub, Streams or Kafka?
| Need | Choose |
|---|---|
| Broadcast to whoever is online, loss OK | Pub/Sub |
| Simple job queue, small scale | List with BRPOP/LMOVE |
| Durable messages, consumer groups, replay, acks | Streams |
| Very high volume, long retention, many independent readers | Kafka (see the Kafka course) |
Streams are bounded by memory, so set a MAXLEN ~ N or MINID trimming policy.
Key takeaways
- Pub/Sub is fire-and-forget: only currently connected subscribers receive messages.
- Streams store entries; consumer groups split work and track unacknowledged entries.
XACKafter processing;XAUTOCLAIMrecovers messages from dead consumers.- Delivery is at-least-once, so make handlers idempotent, and trim streams to bound memory.
Exercises
- Add
maxlen=1000, approximate=TruetoXADDand confirm the stream stays bounded. - Make a worker that retries a message three times, then moves it to a
orders:deadstream. - Use
XRANGEto replay all orders from a given ID, for example to rebuild a report.
Quiz
- What happens to a Pub/Sub message when nobody is subscribed?
- What is the pending entries list?
- Why should stream consumers be idempotent?
Show answers
- It’s dropped;
PUBLISHreturns 0 and nothing stores it. - The per-group list of entries delivered to a consumer but not yet acknowledged with
XACK. - Delivery is at-least-once: an entry processed before a crash but not acknowledged will be delivered again.