Reliability & Graceful Degradation

Redundancy and failover

Failure is arithmetic. A server that fails once every 3 years fails about once per 1,100 days; run 1,000 of them and one fails roughly every day. The bigger the fleet, the more certain it is that something is down right now, so you design so that no single failure matters.

The rule is no single point of failure: for each component, ask what happens if it vanishes, and if the answer is "the service stops," add a second one. N+1 means enough capacity to carry peak with one unit missing: if 4 app servers carry peak, run 5. Multi-AZ puts the copies in separate availability zones (independent power and network), so one fire or fiber cut costs a third of capacity, not all of it. N+1 inside one zone protects against a bad machine and nothing else.

Redundancy helps only if traffic moves off the dead unit. That is failover, and it has two designs.

The mechanism is detect, then reroute. Something notices the failure: a load balancer probing each backend every 2–5 seconds and removing one after 2–3 misses, as the load balancing chapter describes, or an orchestrator watching a database primary's heartbeat. Then something changes where requests go: the balancer drops the node, or a standby is promoted. Detection time is your outage floor. With a 5-second probe and 3 misses, a dead node keeps receiving traffic for up to 15 seconds.

Failover has two classic ways to go wrong.

• Split-brain. The standby cannot reach the primary, concludes it is dead and promotes itself, but the primary is only partitioned away and still accepting writes. Two nodes now accept different writes, and merging them is painful. Require a majority (quorum) to promote, and fence the old primary (revoke its storage lease or network access) before promoting. A fast unfenced failover is more dangerous than a slow one.
• The thundering reconnect. When a primary dies, every app server's pooled connections break at once, and after promotion they all reconnect and retry in the same second. The new primary, with a cold buffer pool, takes a burst it never sees in steady state. Use jittered reconnects and connection-pool ceilings.

Timeouts and retries

Most outages that spread start with a call that never returns. Many client libraries default to no timeout, and the stuck worker is held the whole time. Little's Law says concurrency equals arrival rate times time in system. A service taking 1,000 rps with a 50 ms dependency holds about 50 requests in flight. If the dependency hangs and calls wait 30 seconds, the same rate wants 30,000 in flight, and a pool of 200 workers is full in a fifth of a second. Then even requests that never touch the broken dependency queue for a worker. Set a timeout on every call that leaves the process, a bit above the dependency's p99 as the tail latency chapter measures it, not a round 30 seconds.

A timeout turns a hang into an error, and an error invites a retry. Retries suit brief faults but add load to a system short of capacity. Three rules keep them safe.

Back off exponentially, with jitter. Wait longer after each failure (100 ms, 200 ms, 400 ms…) up to a cap. Without randomness, clients that failed together retry together, and the dependency sees synchronized waves. Full jitter picks the delay at random between zero and the exponential ceiling, which spreads the waves into a trickle.

def call_with_retries(fn, attempts=3, base=0.1, cap=2.0, timeout=0.3):
    for attempt in range(attempts):
        try:
            return fn(timeout=timeout)                  # always bounded
        except (Timeout, Unavailable):
            if attempt == attempts - 1 or not retry_budget.try_spend():
                raise                                   # give up: fail fast
            ceiling = min(cap, base * 2 ** attempt)
            time.sleep(random.uniform(0, ceiling))      # full jitter

Know the amplification. If each of three layers retries 3 times, one user request becomes 3 × 3 × 3 = 27 calls at the bottom. The retries you wrote to survive a blip become a self-inflicted denial of service. Retry in one layer, usually the one closest to the user, and have the others fail fast.

Cap retries with a budget. Three retries per request is still a 4x load when everything fails. A retry budget allows a retry only while retries are under, say, 10% of recent requests. When the dependency is healthy almost nothing is spent; when it fails everywhere the budget runs out and clients stop adding load.

Circuit breakers and bulkheads

Backoff slows each client. A circuit breaker stops them when the evidence says the dependency is down. It wraps calls to one dependency, tracks recent outcomes, and has three states.

• Closed: normal. Calls pass while a sliding window counts failures, for example "more than 50% of the last 20 calls failed."
• Open: calls fail in microseconds without touching the network. The dependency gets silence to recover, and your workers are not held for a timeout each. After a cool-down (say 10 seconds) the breaker moves on.
• Half-open: a few trial calls pass. If they succeed, the breaker closes; if any fails, it reopens for another cool-down. Full traffic at a dependency that just came back is how you knock it over again.

The gain is fail-fast. Without a breaker, every request waits out its 300 ms timeout before failing. With one, the cost is near zero and the caller moves straight to its fallback. A breaker is half a design unless the caller has a fallback, which the degradation section covers.

A bulkhead asks a different question: when one dependency is slow, how much of your capacity can it take with it? Ship hulls use watertight compartments so one breach does not sink the vessel. In software, give each dependency its own pool, a separate connection or thread pool with its own cap. With 200 shared workers and a stalled recommendations API, all 200 can end up waiting on it. With a pool of 20 for recommendations, the other 180 keep serving. The cost is lower utilization: idle capacity in one pool cannot help another.

Rate limiting and load shedding

Every service has a capacity. Above it, arrivals exceed departures and the queue grows without bound, and by Little's Law (chapter 1) waiting time grows with it. A server that does 1,000 rps and receives 1,200 falls 200 requests further behind every second. After 30 seconds the queue holds 6,000 requests and each new arrival waits 6 seconds, by which point most clients have timed out or retried. The server still works, but for people who have left. Goodput, the useful throughput, falls toward zero while the CPU sits at 100%.

The answer is to reject early and cheaply. Refusing costs microseconds and gives the caller a fast, honest error (HTTP 429 or 503, with Retry-After). Accepting a request and failing it after 30 seconds uses the same capacity as serving it and returns nothing. Keep queues short and bounded.

Two mechanisms, with different jobs.

Rate limiting is a policy applied to callers: this API key may make 100 requests per minute. It stops one client, abusive or buggy, taking more than its share. The standard algorithm is the token bucket: a bucket holds up to B tokens and refills at r per second; each request removes one, and with an empty bucket the request is rejected. That allows a burst of B but a sustained rate of at most r. The leaky bucket instead queues requests and drains them at a fixed rate: smooth output, but delay instead of bursts. Prefer the token bucket for APIs, where bursts are normal.

Load shedding is the server's decision about itself: I am overloaded, so I drop some work whoever sent it. Signals include in-flight concurrency over a limit, queue depth, or queueing delay beyond a threshold. Shed selectively: tag requests with a priority and drop the cheapest to lose first, such as background refreshes before user-facing reads, and dashboard reads before the write that completes a purchase.

Graceful degradation and cascading failure

Overload spreads. Service C, a dependency of B, slows after a bad deploy. B's workers wait on C and pile up until B's threads are gone. A's calls to B time out and A's workers pile up. A's callers retry, adding load to a system already behind, and the load balancer, seeing A's health checks time out, pulls A's nodes from rotation, which pushes their traffic onto the rest and makes those fail faster. One slow component is now a full outage, and fixing C does not end it, because the retry storm holds the system down. This is a cascading failure: a positive feedback loop where failure raises load on the survivors, which raises failure.

Every tool in this chapter breaks that loop at a different point. Timeouts bound how long a worker is held. Bulkheads stop the pile-up crossing pools. Breakers stop calls to a sick dependency. Budgets stop retry growth. Shedding keeps the survivors under capacity so they finish the work they accept; without it nothing stops the cascade.

The stance is degrade, don't collapse. For each dependency, decide ahead of time what the service does when it is gone, and write that path as carefully as the happy one.

• Serve stale. If the database struggles, answer from cache even past the TTL. A 10-minute-old answer beats an error.
• Turn off the optional feature. Drop recommendations or live counters and render the core page without them. Use feature flags so an operator can do it in seconds.
• Return a partial result. A search over 20 shards where 2 time out can return the 18 that answered, with a note.

A fallback that has never run in production is a guess, and it often depends on the very thing that failed. Switch the dependency off on purpose, in staging first, then in production at a quiet hour.

Snip under failure

Snip's rule: the redirect is the product, and everything else is optional. At 1,200 rps average and 6,000–12,000 at peak, a redirect must work while much of the rest is on fire.

• The click pipeline is down. Redirects do not wait for it. As in the queues chapter, each click goes to a queue and the redirect returns at once. If the queue is full, the redirect path drops or samples the event rather than block. At 100 B per click, 100 million clicks a day is 10 GB, so an hour of backlog is under 0.5 GB and the queue absorbs a long outage. Past that Snip loses some analytics, never a redirect.
• A shard primary dies. Its replica (chapter 4) is promoted behind a quorum and fencing. Reads for that shard's codes keep working from cache and surviving replicas. Writes (new links) fail for the detection-plus-promotion window, tens of seconds, and the client retries with the same idempotency key.
• Redis is down. Cache-aside (chapter 3) falls through to the database: slower, but correct. The risk is the database meeting the full 1,200–12,000 rps it was sized to see at about 1%. So guard the fall-through with a per-shard concurrency cap, per-key coalescing, an in-process cache for the hottest codes, and shedding of the lowest-priority reads at the cap.
• Abuse. Link creation is rate limited per account and per IP with a token bucket (say 10 links per minute, burst of 20), because each creation is a database write. Code-guessing crawlers hit a per-IP limit on redirect misses; negative caching keeps each bad guess cheap and the limit cuts the guesses off.

The ordering of what gives way, first to last:

1. Click recording and live counters: dropped or delayed first.
2. Dashboards, link statistics, and other reads that are not redirects.
3. Link creation and edits, rejected with a retryable error, because a user can try again.
4. Redirects for cold codes that must hit the database.
5. Redirects for hot, cached codes. This is the last thing to go, and the design should make it nearly impossible.

Writing the hierarchy down is the work. Under pressure nobody argues about what to cut, because the breakers and shed rules already encode it. Keeping enough headroom that shedding rarely triggers is a capacity question for the tail latency and capacity chapter.

✓ Check yourself

Answer from memory before opening each one.

1. A service handles 500 rps with a 40 ms dependency and a pool of 100 workers. The dependency begins to hang with no timeout configured. How quickly does the pool run out, and why do requests that never use that dependency also begin to fail?

   Show answer

   Normal concurrency is 500 × 0.04 = 20 workers. Hung calls never release their workers, so 500 new requests per second consume the 80 free ones in under a quarter of a second. After that, every request, including those that never call the dependency, waits for a free worker. A timeout bounds how long each worker is held. See Timeouts and retries.

2. A colleague adds "retry 3 times" at the web tier, the service tier and the database client. The database slows slightly and then falls over. Explain why, and what you would change.

   Show answer

   Retries multiply across layers: 4 attempts at each of three layers can turn one request into up to 64 database calls, so a modest slowdown becomes a flood. More backoff alone does not remove the multiplication. Retry in one layer only, add jitter, and cap retries with a budget (about 10% of traffic) so a failing dependency stops receiving extra load. See Timeouts and retries.

3. Why does a circuit breaker move to half-open instead of going straight back to closed when its cool-down ends?

   Show answer

   The breaker has no evidence the dependency has recovered; the timer only says time has passed. Full traffic could knock over a dependency that just came back. Half-open sends a few trial calls: success closes the circuit, failure opens it again for another cool-down. See Circuit breakers and bulkheads.

4. An API sits at 100% CPU and its queue holds 8,000 requests, yet it reports almost no successful responses. Why, and why is adding a bigger queue the wrong response?

   Show answer

   Queueing delay is now longer than the client timeout, so the server works on requests whose callers have already given up. Goodput collapses while utilization stays high. A bigger queue lengthens the delay and makes it worse. Bound the queue and reject early, so the work done is for callers still waiting. See Rate limiting and load shedding.

5. A product manager wants every request to the checkout service treated equally during overload "to be fair." What do you propose instead, and when would equal treatment be right?

   Show answer

   Shed by priority: drop background and read-only traffic first and protect the requests that complete a purchase, because losing them costs revenue and a retry later costs little. Equal treatment is right when requests are interchangeable, or when clients mark everything high and the signal cannot be trusted. See Graceful degradation.

6. Redis fails during Snip's peak of 10,000 rps. The design says cache-aside makes this a slowdown. What can still go wrong, and which controls prevent it?

   Show answer

   The database, sized for about 1% of traffic, now receives all of it and may collapse, turning a slowdown into an outage. Prevent it with a concurrency cap on database reads, per-key request coalescing, an in-process cache for the hottest codes, and shedding of low-priority reads once the cap is reached. See Snip under failure.

Takeaway

• At scale, failure is constant, so a design is judged by what happens when a component dies: remove every single point of failure with N+1 capacity spread across zones.
• Failover is detect, then reroute, and detection time is your outage floor; unfenced promotion risks split-brain and mass reconnects can knock over the new primary.
• Every outbound call needs a timeout, because an unbounded wait ties up a worker and, by Little's Law, exhausts the pool within a second.
• Retry in one layer, with exponential backoff, full jitter and a budget of roughly 10% of traffic, and only for idempotent operations; otherwise retries multiply a small fault into a self-inflicted overload.
• Circuit breakers fail fast and let a dependency recover; bulkheads cap how much of your capacity one slow dependency can consume.
• Above capacity, queues grow without bound and goodput collapses, so reject early and shed low-priority work first; rate limits police callers, shedding protects the server.
• Decide the order in which features give way before the incident: Snip drops click counts first and the cached redirect last. Next, A Worked Design: Snip at Scale puts every piece of this guide into one end-to-end design.

Further reading

• Handling Overload — Google SRE Book. Per-client and adaptive throttling, criticality levels and retry budgets, from a fleet that lives with overload.
• Addressing Cascading Failures — Google SRE Book. Why overload feeds on itself, with the mitigations and the order to apply them.
• Timeouts, retries, and backoff with jitter — Marc Brooker, AWS Builders' Library. The clearest account of retry amplification and why full jitter works.
• CircuitBreaker — Martin Fowler. The three states in a short article, with the monitoring a breaker needs.
• Release It! — Michael Nygard, 2nd ed., 2018. The source of the circuit breaker and bulkhead patterns, written from production outages.