Lesson 6 of 10 Patterns

Redis Transactions, WATCH and Lua Scripts

Atomic operations in Redis: MULTI/EXEC transactions, WATCH for optimistic locking with retries, and a Lua token bucket rate limiter, all in Python.

40-50 min intermediate 0/10 lessons done
Course contents Lesson 6 of 10 · 0 done
On this page (8 sections)
  1. What you will learn
  2. The code
  3. MULTI/EXEC
  4. WATCH: optimistic locking
  5. Lua scripts
  6. Key takeaways
  7. Exercises
  8. Quiz

What you will learn

  • What MULTI/EXEC guarantees and what it doesn’t.
  • Optimistic concurrency with WATCH.
  • Lua scripts for read-decide-write logic that must be atomic.

The code

"""Atomicity in Redis: MULTI/EXEC, WATCH for optimistic locking, and Lua scripts."""
import os
import threading

import redis

r = redis.Redis(port=int(os.getenv("REDIS_PORT", 6379)), decode_responses=True)
r.flushdb()

# MULTI/EXEC: queue commands, run them together with nothing interleaved
pipe = r.pipeline(transaction=True)
pipe.hset("order:1", mapping={"status": "paid", "amount": 499})
pipe.lpush("queue:ship", "order:1")
pipe.incr("stats:orders")
print("exec results:", pipe.execute())

# WATCH: optimistic transfer that retries if a balance changed underneath us
r.set("bal:a", 100)
r.set("bal:b", 0)


def transfer(src, dst, amount):
    with r.pipeline() as p:
        while True:
            try:
                p.watch(src)
                balance = int(p.get(src))
                if balance < amount:
                    p.unwatch()
                    return False
                p.multi()
                p.decrby(src, amount)
                p.incrby(dst, amount)
                p.execute()  # raises WatchError if src changed since WATCH
                return True
            except redis.WatchError:
                continue  # someone else changed it: re-read and retry


threads = [threading.Thread(target=transfer, args=("bal:a", "bal:b", 10)) for _ in range(15)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print("a:", r.get("bal:a"), "b:", r.get("bal:b"), "(never negative, total still 100)")

# Lua: a token bucket that reads and writes atomically on the server
TOKEN_BUCKET = r.register_script("""
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_per_ms = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity
local ts = tonumber(state[2]) or now
tokens = math.min(capacity, tokens + (now - ts) * refill_per_ms)
local allowed = 0
if tokens >= 1 then
  tokens = tokens - 1
  allowed = 1
end
redis.call('HSET', key, 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', key, 60000)
return allowed
""")
now_ms = 1_000_000
burst = [TOKEN_BUCKET(keys=["tb:u1"], args=[3, 0.001, now_ms]) for _ in range(5)]
later = TOKEN_BUCKET(keys=["tb:u1"], args=[3, 0.001, now_ms + 2000])  # 2 s later: 2 tokens refilled
print("burst of 5:", burst, "| 2 s later:", later)

Output:

exec results: [2, 1, 1]
a: 0 b: 100 (never negative, total still 100)
burst of 5: [1, 1, 1, 0, 0] | 2 s later: 1

MULTI/EXEC

Commands sent after MULTI are queued; EXEC runs them all in order with no other client’s commands in between. redis-py’s pipeline(transaction=True) sends them in one round trip.

What it does not do:

  • No rollback: if one command fails at runtime (say INCR on a non-number), the others still run.
  • No reads mid-transaction: queued commands return results only at EXEC, so you can’t branch on a value read inside the transaction.

WATCH: optimistic locking

To branch on a value (“transfer only if the balance is enough”), WATCH the key, read it, then MULTI ... EXEC. If any watched key changed in between, EXEC aborts (redis-py raises WatchError) and you retry. Fifteen threads each tried to move 10 from a to b; ten succeeded, five saw an insufficient balance, and the total stayed 100.

WATCH works well under low contention. Under heavy contention, retries pile up, and a Lua script is better.

Lua scripts

A script runs on the server as a single atomic step: no other command executes while it runs. That makes read-decide-write logic simple and fast (one round trip). The token bucket:

  • Reads tokens and the last timestamp from a hash.
  • Refills tokens for the elapsed time, up to capacity.
  • Takes a token if available, writes the state back, and returns 1 or 0.

With capacity 3, a burst of five gets three through; two seconds later (at 1 token per second) a request is allowed again. The script takes the current time as an argument, which keeps it deterministic and makes it easy to test with fixed times, as in the demo. In production pass the real time, or call redis.call('TIME') inside the script.

register_script sends the script once and then calls it by SHA (EVALSHA), retrying automatically if the server lost its script cache. Redis 7+ also offers functions (FUNCTION LOAD), which persist with the data.

Rules for scripts:

  • Keep them short: a long script blocks the server like any slow command.
  • Pass every key in KEYS (not built inside the script), which Redis Cluster needs to route the call.

Key takeaways

  • MULTI/EXEC runs queued commands together but doesn’t roll back and can’t branch on reads.
  • WATCH adds optimistic concurrency: abort and retry if a watched key changed.
  • Lua scripts make read-decide-write atomic in one round trip; keep them short.
  • Pass keys via KEYS so scripts work in Redis Cluster.

Exercises

  1. Make the transfer a Lua script and compare its throughput with the WATCH version under 50 threads.
  2. Change the token bucket to take n tokens at once (for weighted requests).
  3. Show that MULTI/EXEC doesn’t roll back: queue SET a 1, INCR on a hash key, SET b 2, and inspect the results.

Quiz

  1. Does a failing command inside EXEC undo the earlier ones?
  2. When should you prefer a Lua script over WATCH?
  3. Why must keys be passed through KEYS instead of built inside the script?
Show answers
  1. No. Redis transactions have no rollback; the other commands still run.
  2. When contention is high (WATCH would retry often) or the logic needs several reads and decisions in one atomic step.
  3. Redis Cluster uses the declared keys to route the script to the right node and check they’re in one slot.

Finished this lesson?

Mark it complete to track your progress (saved in this browser).