What you will learn
- How availability numbers translate into allowed downtime.
- The core resilience patterns: redundancy, timeouts, retries, circuit breakers, bulkheads.
- What to monitor, and how SLOs make reliability measurable.
- Disaster recovery basics and a one-page checklist for interviews and design reviews.
Availability in numbers
| Availability | Downtime per year | Per month |
|---|---|---|
| 99% | ~3.65 days | ~7.3 hours |
| 99.9% | ~8.8 hours | ~44 minutes |
| 99.99% | ~53 minutes | ~4.4 minutes |
| 99.999% | ~5 minutes | ~26 seconds |
Components in series multiply: three services each at 99.9%, all required for a request, give about 99.7%. Redundant copies in parallel improve it: two independent 99% servers give 99.99% for “at least one is up”. That arithmetic is why we remove single points of failure.
Resilience patterns
- Redundancy: at least two of everything on the request path, spread across availability zones. Databases replicated with automatic failover (lesson 5).
- Timeouts: every network call gets one. A call without a timeout can hang a thread forever and, under load, the whole service.
- Retries with backoff and jitter: only for idempotent operations, with a cap. Unbounded retries turn a small outage into a retry storm.
- Circuit breaker: after repeated failures to a dependency, stop calling it for a while and fail fast (or serve a fallback), then probe with a few requests before closing again.
- Bulkheads: separate thread pools/connection pools per dependency so one slow dependency can’t consume all resources.
- Graceful degradation: if recommendations are down, show the page without them. Decide in advance what’s essential.
- Load shedding: when overloaded, reject low-priority work early (429/503) to keep the important work fast.
Monitoring and SLOs
Watch the four golden signals for every service: latency (p50, p99), traffic (requests/s), errors (rate), and saturation (CPU, memory, queue depth, connection pools).
An SLI is a measurement (“fraction of redirects served under 50 ms”). An SLO is a target for it (“99.9% over 30 days”). The gap to 100% is the error budget: while you have budget left, ship features; when it’s spent, focus on reliability. Alert on SLO burn rate, not on every CPU spike.
Add distributed tracing (a trace id passed through every service) to see where time goes in a request, and structured logs with that trace id.
Disaster recovery
Two numbers frame it: RPO (how much data can you lose? e.g. 5 minutes) and RTO (how long to recover? e.g. 1 hour). Backups must be tested by actually restoring them. Multi-region setups range from cold standby (cheap, slow RTO) to active-active (expensive, near-zero RTO, but you must handle cross-region consistency).
The checklist
Use this in interviews and design reviews:
- Requirements: core features (2–4), explicit non-goals, numbers for scale, latency, availability, consistency.
- Estimates: QPS average and peak, read/write ratio, storage over time, bandwidth.
- API and data model: endpoints, entities, access patterns.
- High-level design: request path for each core flow; every box justified.
- Data: SQL vs NoSQL, indexes, replication, shard key, caching strategy.
- Async: what goes through queues; delivery guarantees; idempotency.
- Scale: stateless tiers behind load balancers, CDN, hot keys, celebrity/fan-out issues.
- Failure: what happens when each component dies; timeouts, retries, failover.
- Operations: metrics, SLOs, alerts, deploys, DR.
- Trade-offs: say what you chose, what you rejected, and why.
Then practise: work through the prompts on the HLD practice sheet using this list, and compare your designs with the case studies in lessons 10–14.
Key takeaways
- Every extra nine cuts allowed downtime by 10x; series dependencies multiply unavailability, redundancy divides it.
- Timeouts, bounded retries with jitter, circuit breakers, bulkheads and load shedding stop failures from spreading.
- Monitor latency, traffic, errors and saturation; manage reliability with SLOs and error budgets.
- Define RPO and RTO and test restores; walk the checklist for every design.
Exercises
- A request path has a load balancer (99.99%), app tier (99.95%), cache (99.9%, but with database fallback) and database (99.95%). Estimate the end-to-end availability.
- Implement a minimal circuit breaker class in Python with closed, open and half-open states.
- Write three SLOs for the URL shortener from lesson 10, with the SLI behind each.
Quiz
- Roughly how much downtime per month does 99.9% allow?
- Why should retries only be used for idempotent operations?
- What’s the difference between RPO and RTO?
Show answers
- About 44 minutes.
- A retried non-idempotent operation (like a payment) can apply twice if the first attempt actually succeeded.
- RPO is how much data (measured in time) you can afford to lose; RTO is how long recovery may take.