What you will learn
- Where caches sit in a system and what makes a good cache candidate.
- The four caching patterns and when to use each.
- Eviction policies, TTLs and how to invalidate safely.
- The failure modes: stampede, hot keys, stale data, cold start.
Why cache
From lesson 2: RAM is roughly 1,000x faster than SSD and a network hop to a database costs ~0.5 ms plus query time. If 90% of reads hit a cache, the database sees one tenth of the load and most users get sub-millisecond responses.
Good candidates are read-heavy, expensive to compute and tolerant of slight staleness: user profiles, product pages, rendered feeds, results of heavy queries. Bad candidates change constantly or must always be exact (an account balance during a payment).
Caches exist at many layers: browser, CDN, load balancer, application memory, a shared cache like Redis or Memcached, and the database’s own buffer pool.
Caching patterns
Cache-aside (lazy loading)
The application checks the cache; on a miss it reads the database and fills the cache. On a write it updates the database and deletes the cache entry.
"""Cache-aside (lazy loading) with a TTL, in front of a slow 'database'."""
import time
class SlowDatabase:
def __init__(self):
self.rows = {"user:1": {"name": "Asha"}, "user:2": {"name": "Ravi"}}
self.reads = 0
def get(self, key):
self.reads += 1
time.sleep(0.01) # pretend this is a 10 ms query
return self.rows.get(key)
def put(self, key, value):
self.rows[key] = value
class TTLCache:
def __init__(self, ttl_seconds: float):
self.ttl = ttl_seconds
self.items = {} # key -> (value, expires_at)
def get(self, key):
entry = self.items.get(key)
if entry is None:
return None
value, expires_at = entry
if time.monotonic() > expires_at:
del self.items[key] # expired: treat as a miss
return None
return value
def set(self, key, value):
self.items[key] = (value, time.monotonic() + self.ttl)
def delete(self, key):
self.items.pop(key, None)
def read_user(cache, db, key):
value = cache.get(key)
if value is not None:
return value # cache hit
value = db.get(key) # cache miss: go to the source of truth
if value is not None:
cache.set(key, value)
return value
def update_user(cache, db, key, value):
db.put(key, value) # 1. write the database
cache.delete(key) # 2. invalidate, so the next read reloads fresh data
if __name__ == "__main__":
db, cache = SlowDatabase(), TTLCache(ttl_seconds=60)
for _ in range(100):
read_user(cache, db, "user:1")
print("100 reads, database reads:", db.reads) # 1
update_user(cache, db, "user:1", {"name": "Asha K"})
print("after update:", read_user(cache, db, "user:1"), "database reads:", db.reads) # 2
100 reads, database reads: 1
after update: {'name': 'Asha K'} database reads: 2
It’s the most common pattern: only requested data is cached and the cache can fail without losing data. Why delete instead of updating the cache on writes? Two concurrent writers can update the cache in the opposite order from the database, leaving it wrong until the TTL. Deleting forces the next reader to fetch the truth.
Read-through
Like cache-aside, but the cache library itself loads from the database on a miss. Cleaner application code; same behaviour.
Write-through
Every write goes to the cache and the database synchronously. Reads are always warm and consistent, writes are slower, and you cache data nobody may read.
Write-back (write-behind)
Writes go to the cache and are flushed to the database later in batches. Very fast writes, but data is lost if the cache dies before flushing. Used for counters, metrics and analytics where losing a few seconds is acceptable.
| Pattern | Read latency | Write latency | Risk |
|---|---|---|---|
| Cache-aside | Miss on first read | DB speed | Brief staleness |
| Write-through | Always warm | Slower (two writes) | Caches unused data |
| Write-back | Always warm | Fastest | Data loss on crash |
Eviction and TTL
Caches are smaller than the data, so something must be evicted:
- LRU (least recently used): the default almost everywhere. You’ll implement one in the LLD course.
- LFU (least frequently used): better when popularity is stable.
- TTL (time to live): entries expire after a fixed time, bounding staleness even if invalidation is missed.
Add jitter to TTLs (e.g. 300 s ± 10%) so entries created together don’t all expire together.
Failure modes
Cache stampede (thundering herd). A hot key expires and thousands of requests miss at once and hammer the database. Fixes: let one request rebuild while others wait or get the stale value (a lock or “single flight”), refresh hot keys before they expire, and use jittered TTLs.
Hot keys. One key (a celebrity profile, a viral link) gets so much traffic that one cache node melts. Fixes: replicate the hot key across nodes (with a random suffix), or keep a small in-process cache in front of the shared one.
Stale data. Invalidation messages get lost or race. Keep TTLs as a safety net, and for critical data read from the database.
Cold start. A fresh or flushed cache sends all traffic to the database. Warm it up gradually, or shift traffic slowly.
Cache penetration. Requests for keys that don’t exist always miss. Cache the “not found” result briefly, or use a Bloom filter.
To practise these patterns hands-on with a real cache, see the Redis course.
Key takeaways
- Cache read-heavy, expensive, staleness-tolerant data; RAM beats every disk and network hop.
- Cache-aside with delete-on-write is the safe default; write-through trades write speed for warm reads; write-back trades safety for speed.
- LRU plus TTLs (with jitter) bound memory and staleness.
- Plan for stampedes, hot keys, stale entries, cold starts and lookups of missing keys.
Exercises
- Add hit/miss counters to
TTLCacheand print the hit ratio for 1,000 reads following a skewed (Zipf-like) distribution of 100 keys. - Add single-flight protection: when a key is being loaded, other callers wait for that load instead of querying the database.
- Describe a sequence of two concurrent writers and one reader in which “update the cache on write” leaves the cache permanently wrong (until TTL).
Quiz
- In cache-aside, what happens on a cache miss?
- Why is write-back risky?
- What is a cache stampede and one way to prevent it?
Show answers
- The application reads from the database, stores the result in the cache, and returns it.
- Writes are acknowledged once they’re in the cache; if the cache fails before flushing to the database, those writes are lost.
- Many requests miss the same expired hot key simultaneously and overload the database. Prevent it with a rebuild lock / single flight, early refresh, or jittered TTLs.