What you will learn
- What
MULTI/EXECguarantees and what it doesn’t. - Optimistic concurrency with
WATCH. - Lua scripts for read-decide-write logic that must be atomic.
The code
"""Atomicity in Redis: MULTI/EXEC, WATCH for optimistic locking, and Lua scripts."""
import os
import threading
import redis
r = redis.Redis(port=int(os.getenv("REDIS_PORT", 6379)), decode_responses=True)
r.flushdb()
# MULTI/EXEC: queue commands, run them together with nothing interleaved
pipe = r.pipeline(transaction=True)
pipe.hset("order:1", mapping={"status": "paid", "amount": 499})
pipe.lpush("queue:ship", "order:1")
pipe.incr("stats:orders")
print("exec results:", pipe.execute())
# WATCH: optimistic transfer that retries if a balance changed underneath us
r.set("bal:a", 100)
r.set("bal:b", 0)
def transfer(src, dst, amount):
with r.pipeline() as p:
while True:
try:
p.watch(src)
balance = int(p.get(src))
if balance < amount:
p.unwatch()
return False
p.multi()
p.decrby(src, amount)
p.incrby(dst, amount)
p.execute() # raises WatchError if src changed since WATCH
return True
except redis.WatchError:
continue # someone else changed it: re-read and retry
threads = [threading.Thread(target=transfer, args=("bal:a", "bal:b", 10)) for _ in range(15)]
for t in threads:
t.start()
for t in threads:
t.join()
print("a:", r.get("bal:a"), "b:", r.get("bal:b"), "(never negative, total still 100)")
# Lua: a token bucket that reads and writes atomically on the server
TOKEN_BUCKET = r.register_script("""
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_per_ms = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity
local ts = tonumber(state[2]) or now
tokens = math.min(capacity, tokens + (now - ts) * refill_per_ms)
local allowed = 0
if tokens >= 1 then
tokens = tokens - 1
allowed = 1
end
redis.call('HSET', key, 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', key, 60000)
return allowed
""")
now_ms = 1_000_000
burst = [TOKEN_BUCKET(keys=["tb:u1"], args=[3, 0.001, now_ms]) for _ in range(5)]
later = TOKEN_BUCKET(keys=["tb:u1"], args=[3, 0.001, now_ms + 2000]) # 2 s later: 2 tokens refilled
print("burst of 5:", burst, "| 2 s later:", later)
Output:
exec results: [2, 1, 1]
a: 0 b: 100 (never negative, total still 100)
burst of 5: [1, 1, 1, 0, 0] | 2 s later: 1
MULTI/EXEC
Commands sent after MULTI are queued; EXEC runs them all in order with no other client’s commands in between. redis-py’s pipeline(transaction=True) sends them in one round trip.
What it does not do:
- No rollback: if one command fails at runtime (say
INCRon a non-number), the others still run. - No reads mid-transaction: queued commands return results only at
EXEC, so you can’t branch on a value read inside the transaction.
WATCH: optimistic locking
To branch on a value (“transfer only if the balance is enough”), WATCH the key, read it, then MULTI ... EXEC. If any watched key changed in between, EXEC aborts (redis-py raises WatchError) and you retry. Fifteen threads each tried to move 10 from a to b; ten succeeded, five saw an insufficient balance, and the total stayed 100.
WATCH works well under low contention. Under heavy contention, retries pile up, and a Lua script is better.
Lua scripts
A script runs on the server as a single atomic step: no other command executes while it runs. That makes read-decide-write logic simple and fast (one round trip). The token bucket:
- Reads
tokensand the last timestamp from a hash. - Refills tokens for the elapsed time, up to capacity.
- Takes a token if available, writes the state back, and returns 1 or 0.
With capacity 3, a burst of five gets three through; two seconds later (at 1 token per second) a request is allowed again. The script takes the current time as an argument, which keeps it deterministic and makes it easy to test with fixed times, as in the demo. In production pass the real time, or call redis.call('TIME') inside the script.
register_script sends the script once and then calls it by SHA (EVALSHA), retrying automatically if the server lost its script cache. Redis 7+ also offers functions (FUNCTION LOAD), which persist with the data.
Rules for scripts:
- Keep them short: a long script blocks the server like any slow command.
- Pass every key in
KEYS(not built inside the script), which Redis Cluster needs to route the call.
Key takeaways
MULTI/EXECruns queued commands together but doesn’t roll back and can’t branch on reads.WATCHadds optimistic concurrency: abort and retry if a watched key changed.- Lua scripts make read-decide-write atomic in one round trip; keep them short.
- Pass keys via
KEYSso scripts work in Redis Cluster.
Exercises
- Make the transfer a Lua script and compare its throughput with the WATCH version under 50 threads.
- Change the token bucket to take
ntokens at once (for weighted requests). - Show that
MULTI/EXECdoesn’t roll back: queueSET a 1,INCRon a hash key,SET b 2, and inspect the results.
Quiz
- Does a failing command inside
EXECundo the earlier ones? - When should you prefer a Lua script over WATCH?
- Why must keys be passed through
KEYSinstead of built inside the script?
Show answers
- No. Redis transactions have no rollback; the other commands still run.
- When contention is high (WATCH would retry often) or the logic needs several reads and decisions in one atomic step.
- Redis Cluster uses the declared keys to route the script to the right node and check they’re in one slot.