What you will learn
- When to scale up and when to scale out.
- Why stateless application servers are the foundation of horizontal scaling.
- How load balancers work: layers, algorithms, health checks.
- Where CDNs and autoscaling fit in.
Scale up or scale out
Vertical scaling (scale up) means a bigger machine: more CPU, RAM, faster disks. It’s simple (no code changes) and often the right first step. Limits: there’s a biggest machine, prices rise steeply near the top, and one machine is still one point of failure.
Horizontal scaling (scale out) means more machines sharing the work. It has no hard ceiling and survives individual failures, but needs a way to spread requests (a load balancer) and a design where any machine can serve any request.
Most real systems do both: scale the database up as far as is sensible, scale the stateless tiers out.
Stateless services
A server is stateless if it keeps nothing between requests that another server would need. All durable state lives in shared stores: databases, caches, object storage.
+-> app-1 --+
client -> LB +-> app-2 --+--> database / cache / object storage
+-> app-3 --+
Then any request can go to any server, you can add or remove servers at will, and a crashed server loses nothing. Typical state to move out: user sessions (to Redis or a signed token such as a JWT), uploaded files (to object storage like S3), in-memory counters (to a shared store).
Sticky sessions (the load balancer always sends a user to the same server) are a workaround for stateful servers. They make balancing uneven and lose sessions when a server dies. Prefer making the servers stateless.
Load balancers
A load balancer (LB) receives client connections and forwards each request to a healthy backend.
| Layer 4 (transport) | Layer 7 (application) | |
|---|---|---|
| Sees | IP addresses and ports | Full HTTP: path, headers, cookies |
| Can route by | Connection | URL, host, header, user |
| Extra features | Very fast, protocol-agnostic | TLS termination, compression, retries, A/B routing |
| Examples | AWS NLB, LVS | Nginx, HAProxy, Envoy, AWS ALB |
Balancing algorithms
- Round robin: next server in turn. Good default when servers and requests are uniform.
- Weighted round robin: bigger servers get more.
- Least connections: pick the server with fewest active connections. Good when requests vary in length.
- Hash-based (by client IP or user id): the same key lands on the same server; useful for cache locality. Use consistent hashing (lesson 6) so adding a server doesn’t reshuffle everyone.
- Power of two choices: pick two servers at random, choose the less loaded. Nearly as good as least-connections with far less coordination.
Health checks
The LB periodically calls something like GET /healthz on each backend and stops sending traffic to servers that fail. Make health checks meaningful (can the server reach its database?) but cheap, and distinguish “alive” from “ready to serve”.
The LB is not a single point of failure
Run load balancers in pairs (active-passive with a floating IP) or as a managed, distributed service. At the top level, DNS can return several LB addresses, and GeoDNS / anycast sends users to the nearest region.
CDNs
A content delivery network caches static (and some dynamic) content on edge servers close to users. Images, video, JS and CSS are served from a city near the user instead of your origin, cutting latency from ~150 ms to ~10–20 ms and removing most of the load from your servers. Cache-control headers decide what the CDN may keep and for how long.
Autoscaling
With stateless servers behind a load balancer, you can add and remove instances automatically based on CPU, request rate or queue depth. Keep a minimum for baseline traffic, scale out early (new instances take time to boot and warm up), and scale in slowly.
Key takeaways
- Scale up first for simplicity; scale out for capacity and fault tolerance.
- Stateless app servers (state in shared stores) make horizontal scaling trivial.
- L4 balancers are fast and simple; L7 balancers route on HTTP details and terminate TLS.
- Health checks remove bad servers; redundancy removes the LB as a single point of failure; CDNs push content to the edge.
Exercises
- List every piece of state a typical login-based web app keeps in server memory, and where you’d move each one.
- Simulate round robin vs least connections in a few lines of Python with requests of random duration. Compare the maximum load on any server.
- Design the health check endpoint for a service that depends on a database and a cache. What should it return when the cache is down?
Quiz
- Why do sticky sessions hurt scalability?
- Which load balancer type can route
/api/*and/images/*to different backends? - What two problems does a CDN solve?
Show answers
- Load can’t be spread freely (popular users pin servers), and when a server dies its users’ sessions are lost.
- A layer 7 (HTTP-aware) load balancer.
- Latency for distant users (content is served nearby) and load on the origin servers.