Load Balancing & Statelessness

Why statelessness comes first

Put two copies of a server behind a balancer and you have made a bet: that either copy can answer any request. The bet holds only if a request carries, or can fetch from a shared place, everything the server needs. Once a server keeps something in its own process that a later request depends on, the bet fails. Server A remembers the login; server B has never heard of it, and the balancer can no longer route freely.

So a stateless worker is one whose answer depends only on the request and on stores shared by every worker. The state still exists. It moves to one of two places:

• A shared store. A session ID in a cookie, with the session itself in Redis or the database. Every worker looks it up; none owns it. The cost is a round trip per request and a store that must itself scale.
• The client. A signed token (a JWT, or any HMAC-signed blob) that holds the user ID and expiry. The worker verifies the signature with a shared key and needs no lookup. The cost is that you cannot revoke a token before it expires without a shared denylist, which puts a store back in the path.

The state that bites is the kind nobody labeled as state:

• In-memory sessions that work perfectly with one server and log users out at random with two.
• Local disk. An upload written to /tmp on one node and read back from another that does not have it. Use object storage.
• In-process rate limits. A limit of 100 a minute counted in each process becomes 100 × N across N workers, and shifts every time you scale. The counter belongs in a shared store (chapter 03 covers that kind).
• Scheduled jobs inside the app. A cron-style timer in the service runs on every replica, so the nightly cleanup runs N times. Run it once, from a scheduler or with a lock.

What a load balancer does

A load balancer holds a list of healthy backends, accepts connections on one address, and forwards each to a backend it chooses. It spreads load, hides the failure of any one backend, and gives you one place to add capacity. The first decision is which layer it works at.

The subtle consequence is connection reuse. With HTTP keep-alive or HTTP/2, one client holds one long connection and sends thousands of requests down it. An L4 balancer picks a backend once, so all of that client's requests land on the same server, and a few busy clients can skew the load badly. An L7 balancer sees each request and can spread them. Most web services want L7 at the front.

Then the balancer needs a rule for picking among healthy backends:

Start with round-robin, and move to least-connections the day your p99 shows that some requests are much slower than the rest. Hashing is a deliberate act: you are choosing to bind keys to servers, which is the opposite of statelessness, so see session affinity before you do it.

Health checks and ejection

A balancer that keeps sending traffic to a dead node turns one failure into a share of failed requests equal to 1/N. Health checks prevent that, in two forms. Active checks have the balancer probe each backend on a timer (an HTTP GET /healthz every few seconds) and mark it down after several consecutive failures. Passive checks watch real traffic: if a backend returns connection errors or 5xx responses, count them and eject it for a while. Active checks find a dead node that has no traffic. Passive checks find a node that passes the probe but fails real requests. In open-source NGINX the built-in checks are passive (max_fails, fail_timeout); active probing is a commercial feature there, while Envoy has both (active health checking and outlier detection).

The dangerous part is what the probe asks. If /healthz only returns 200, it proves the process is alive and nothing else. If it also queries the database, it ties every node's health to a shared dependency. Now picture the database getting slow. Every node's probe times out at roughly the same moment; the balancer ejects all of them; and with zero healthy backends it either returns errors to everyone or sends traffic to ejected nodes anyway. The database was slow, not down, and the health checks converted a degraded system into a total outage.

The same machinery handles deploys. To replace a node, first mark it as draining: the balancer stops sending it new requests but lets in-flight ones finish, up to a timeout. Then you stop it. Killing a process mid-request gives a burst of 502s on every release.

Session affinity

Session affinity (sticky sessions) makes the balancer send one client to the same backend repeatedly. It works by cookie (the balancer sets one naming the backend) or by hashing the client IP. It patches the in-memory-session problem: keep the user on the server that holds the state.

It hurts at scale for three reasons that share a root. First, load goes uneven: a few heavy users or one office behind a NAT address pin to one node, and the balancer cannot move them without breaking their sessions, so you cannot even out the load. Second, state dies with the node: when that backend crashes or is drained, every user stuck to it loses their session, so a routine deploy logs people out. Third, new nodes receive only new sessions, so existing hot nodes stay hot.

The recommendation is to be stateless and use no affinity. The exception is cache warmth. If each node holds a large local cache that is expensive to rebuild, for example a per-user model or a big in-memory index, then sending the same key to the same node can raise the hit rate a great deal. Do that with consistent hashing on the cache key, not with a cookie, accept that it is an optimization (a miss must still produce the right answer), and make sure a lost node costs you speed and not correctness.

Snip in this light

Snip's redirect path is a pure function of the request: take the 7-character code, look up the URL, return a 301 or 302 with a Location header. It stores nothing per user, writes no session, and does not care which replica answered. That is why it scales out so cleanly,. The one thing a redirect does besides the lookup, recording the click, is not state the next request needs; it is an event sent onward (chapter 06), so it does not break the property.

The request path from the edge in looks like this:

The load is easy to size. Average traffic is about 1,200 requests per second, with peaks of 5 to 10 times that, so call it 6,000 to 12,000 rps. A small replica doing a cache lookup can plausibly serve a couple of thousand rps (measure yours; this is an assumption), so about 6 replicas cover the peak. Add headroom so losing or draining one leaves the rest under 70% busy, and you land near 8 to 10. Going from 10 to 20 is a config change with no data movement.

Above the balancer sits DNS. Geo-DNS returns the nearest region's address, and anycast announces the same IP from many locations so the network routes each client to the closest one. That balances across regions, but resolvers cache DNS answers for their TTL, so it is coarse and slow to steer; the balancer makes the per-request choice below it. The cost now sits in the lookup, where caching and the database come in. The sizing method is from chapter 01.

✓ Check yourself

Answer from memory before opening each one.

1. A service stores login sessions in each process's memory. It works in staging with one instance. In production with four, users report being logged out every few clicks. What is happening, and name two different fixes with the cost of each?

   Show answer

   The balancer sends successive requests to different instances, and only the one that created the session knows it. Fix one: move sessions to a shared store such as Redis; the cost is a lookup per request and a store you must keep up. Fix two: a signed token the client carries; the cost is that you cannot revoke it early without a shared denylist. See why statelessness comes first.

2. A browser client opens one HTTP/2 connection and sends 5,000 requests over it. Why might an L4 balancer send all 5,000 to one backend, and what changes with L7?

   Show answer

   L4 chooses a backend per connection, not per request, so every request on that connection follows it. An L7 balancer parses each request and can pick a backend for each one. This is why a few busy long-lived clients can skew load behind an L4 balancer. See what a load balancer does.

3. The database behind a 12-node service slows to 8-second queries. Each node's /healthz runs a quick query with a 2-second timeout and the balancer ejects after 3 failures. Describe what the users see and what you would change.

   Show answer

   All 12 nodes fail the probe within seconds of each other and get ejected, so users see errors for the whole service, though the database was only slow. Nodes that would have served cached or degraded answers are removed too. Make the liveness probe local and cheap, cap the share of hosts the balancer may eject, and handle the slow dependency in the request path with timeouts and fallbacks. See health checks.

4. Your team proposes sticky sessions to keep a 4 GB per-user lookup structure warm on each node. When is that sound, and what would you do differently from a cookie?

   Show answer

   It is sound when the structure is a cache: a miss still gives a correct answer, only slower, and rebuilding is expensive enough that locality pays. Route by consistent hashing on the user ID rather than a cookie, so removing a node remaps only about 1/N of users. It is unsound if the structure holds the only copy of anything. See session affinity.

Takeaway

• Horizontal scaling works only when any worker can serve any request, which means no request-relevant state in the process.
• Statelessness moves state to a shared store or into a signed client token, and each choice has a price: a lookup per request, or no early revocation.
• The hidden state to hunt for is in-memory sessions, local disk, per-process rate limits, in-app scheduled jobs and caches that change the answer.
• L4 balancers route per connection and L7 balancers route per request, so long-lived connections skew load behind L4.
• Start with round-robin, use least-connections when request times vary, and use consistent hashing only to bind keys to nodes on purpose.
• Health probes should be cheap and local; a probe that checks a shared dependency can eject every node when that dependency slows, so cap ejection and drain nodes before deploys.
• Prefer no session affinity; the exception is a local cache whose misses cost speed and not correctness, routed by consistent hashing.

The redirect tier now scales by adding replicas, which pushes the load onto the lookup behind it; the next move is to keep most lookups from reaching the database at all. → Caching

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann, O'Reilly, 2017. The scalability and shared-nothing discussion, and the best treatment of why state placement drives architecture.
• The Twelve-Factor App: Processes — Heroku. The short, direct statement of stateless processes with state in backing services.
• NGINX: Using nginx as HTTP load balancer — the algorithms, passive health checks and session persistence as one real balancer implements them.
• Envoy: Load balancing — least-request, ring hash and Maglev, panic threshold, and how ejection limits work. Pair with its health checking and outlier detection pages.
• Consistent Hashing and Random Trees — Karger et al., STOC 1997. The original paper behind consistent hashing; read the introduction for why hash % N fails.