Section C · Consistency, flow & failure

Consistency & Its Price

Once data lives in more than one place (replicas, a cache, shards), the copies disagree for a while. Consistency is the contract about what a read may see during that while, and every stronger contract costs latency and availability. This chapter explains the models, the quorum arithmetic and consensus, then spends them on Snip only where they earn their cost.

What consistency means

Snip now keeps the same link in several places: a primary database, its replicas, a Redis entry and a CDN edge. A write lands on one of them first and travels to the rest over a network that takes time. For that interval the copies hold different values. A reader who asks a copy that has not heard yet gets the old answer. That is the whole problem. Everything else in this chapter is a rule about which old answers a reader may be given.

Take a user who edits a link's destination. The write commits on the primary at time 0. A replica receives it at 40 ms, another at 900 ms because it was busy replaying a backlog. If the user reloads at 100 ms and the load balancer sends the read to the second replica, the edit appears to have vanished. Nothing failed; that is asynchronous replication working as designed.

Consistency, in this chapter, is a guarantee about what reads observe when data is replicated. It is a contract between the storage system and the code that calls it. It is not the "C" in ACID. ACID consistency means a transaction takes the database from one valid state to another, with constraints such as foreign keys and unique indexes intact; it concerns a single database's rules and is enforced by the application and isolation levels. The two share a word and little else.

CAP and PACELC

The CAP theorem is the most quoted and most misread result in the field. Its claim is narrow. Suppose a network partition splits your replicas into two groups that cannot reach each other. A client writes to one side and another client reads from the other. The reader's replica can either answer with what it has, which may be stale, or refuse to answer until it can confirm it is current. Answering keeps the system available and gives up linearizability, the strongest consistency model, defined in the next section. Refusing keeps linearizability and gives up availability. There is no third option, because the information the reader needs is on the far side of the break.

"Pick two of three" misreads this in three ways. Partitions are not a feature you can decline; they happen to you, so the real choice is only between consistency and availability, and only while one is happening. CAP says nothing about normal operation, when there is no partition. And "C" and "A" mean specific, strict things: linearizability, and every non-failing node answering every request. A system that is slow, or that answers only from a majority side, is "unavailable" to CAP yet may be perfectly useful.

Partitions are rare enough that the everyday cost is elsewhere, and Daniel Abadi's PACELC captures it. If there is a partition (P), choose availability (A) or consistency (C); else (E), choose latency (L) or consistency (C). The second half applies to every request you serve. To give a strongly consistent answer, a replica must confirm it is current, which means coordinating with another node. Coordination within a data center costs about 0.5 ms for a round trip; across regions it costs 60–150 ms. A read served from the nearest replica without asking anyone is fast and may be stale. A read that checks with the primary or a majority is correct and slower, by exactly one round trip. That latency is the price of consistency on a healthy day.

The spectrum of models

A consistency model is a rule that forbids some set of surprising results. Stronger models forbid more and force more coordination before a node may answer. This table runs from strongest to weakest.

Model	What it forbids	What it costs
Linearizable (strong)	Any read that returns an older value than a write that has already completed. The system behaves like one copy, and operations take effect at a single instant between their start and finish.	Every read and write coordinates with a leader or majority. Unavailable on the minority side of a partition. One round trip or more per operation.
Sequential	Two clients seeing operations in different orders. All agree on one order that respects each client's own sequence, but it need not match real time.	Still needs a single agreed order, so most of linearizability's coordination; saves little in practice.
Causal	Seeing an effect before its cause: a reply before the post it answers. Unrelated writes may appear in different orders on different nodes.	Track dependencies and delay delivery until causes arrive. Can stay available during a partition, which linearizability cannot.
Session guarantees	Read-your-writes: you never miss your own write. Monotonic reads: you never see data go backward in time.	Cheap. Pin a session to one replica, or carry a version token and have a replica wait until it has caught up to it.
Eventual	Nothing in the short term. Only requires that if writes stop, all replicas end with the same value.	No coordination on reads. Lowest latency, highest availability, and the application sees every anomaly.

Three points in that table deserve a second look. First, linearizable is a statement about real time: once a write returns, no later read anywhere may return the old value. That is what a lock, a unique username or a bank balance needs, and it is almost nothing else. Second, causal is the strongest model that survives a partition. Third, the session guarantees are the practical middle. They give a user a coherent story about their own actions without a global order, and they cost one routing decision. Most product bugs reported as "the system lost my change" are a missing read-your-writes guarantee, not a need for linearizability.

Monotonic reads fixes a different surprise. A user refreshes a list and sees a new link, refreshes again and it is gone, because the second request reached a lagging replica. Sending each user's reads to the same replica removes it without touching the write path.

Quorums: tunable consistency without a leader

Some stores, in the style of Amazon's Dynamo (Cassandra, Riak, ScyllaDB), have no single primary. Each key is stored on N replicas, say N = 3, and any replica may accept a write. The client or a coordinating node sends each write to all N and waits for W of them to acknowledge. A read asks R replicas and takes the newest version among the R answers. The rule that makes this work is one line of arithmetic.

If W + R > N, every read set overlaps every write set in at least one replica. With N = 3, W = 2, R = 2, a write reaches two nodes and a read asks two of three, so at least one node in the read has the latest write. The reader compares version numbers or timestamps and picks the newest. The system never needs to know which nodes are ahead; the overlap guarantees at least one is.

N / W / R	W + R	Behavior
3 / 2 / 2	4 > 3	Overlap. Reads see the latest acknowledged write. Tolerates one slow or dead node for reads and for writes.
3 / 3 / 1	4 > 3	Overlap. Fast reads, but one dead node blocks all writes.
3 / 1 / 1	2 < 3	No overlap. Fastest and most available; reads can be stale for as long as repair takes.

The dial moves latency and availability in opposite directions, which is PACELC as a parameter. A larger W makes writes wait for slower replicas and makes them fail when too many are down. A smaller R makes reads fast and risks stale answers. The common choice, W = R = quorum (2 of 3), tolerates one failure on each path and keeps the overlap.

Overlap is weaker than it sounds. It guarantees that a read can see the latest acknowledged write, not that the system is linearizable. Concurrent writes to the same key can interleave, a write that fails partway may be visible to some readers and not others, and "newest" depends on clocks or version vectors that can pick a loser.

Sloppy quorums and hinted handoff. A strict quorum requires W acknowledgments from the N replicas that own the key. If two are unreachable, writes fail. A sloppy quorum accepts acknowledgments from any W reachable nodes, even ones that do not own the key. A stand-in node holds the write with a "hint" naming the intended owner, and hands it over when the owner returns. This buys write availability through failures and partitions: the write succeeds. It costs the overlap guarantee. A read of the true owners can miss a write sitting on a stand-in, so W + R > N no longer promises fresh reads until handoff completes. You have chosen availability over consistency, openly, per write.

Consensus in one page

Quorums let replicas disagree and patch it up later. Some jobs cannot tolerate that: exactly one node may be leader, exactly one client may hold a lock, a configuration change must take effect for everyone in the same order. These need consensus: a group of nodes agreeing on a value, even though some of them crash and messages are delayed.

Raft and Paxos both solve this by building a replicated log, an ordered list of commands that every node holds and applies in order. If all nodes apply the same commands in the same sequence, they all reach the same state. Agreement on one total order is the product. Raft makes it understandable by splitting the job into parts.

A leader. At any time one node is leader for a numbered term. Clients send writes to it. If followers stop hearing from it, one starts an election for a higher term.
A majority vote. A candidate wins with votes from more than half the nodes. Two candidates cannot both win a term, because two majorities of the same cluster always share a node.
Commit by majority. The leader appends a command to its log, sends it to followers, and counts it committed once a majority has stored it. Then it replies to the client. A committed entry survives any minority of failures, because any future leader must have been elected by a majority that includes a node holding it.

The arithmetic is the same overlap as in quorums, applied with strict rules. A cluster of 3 survives 1 failure; 5 survives 2; 7 survives 3. Past 5 nodes you mostly add cost and slower commits. If a partition leaves a minority, that side cannot elect a leader or commit, so it stops accepting writes. That is the CAP choice, made for consistency.

Where to use it. Leader election; cluster metadata such as which shard lives on which server (the 1,024-shard map from chapter 4 is a good candidate); configuration; distributed locks and leases; and the rare small dataset that must be linearizable. You rarely run it yourself: etcd, ZooKeeper and Consul package it.

Where not to. Do not push high-volume data through one consensus group. Every write pays a leader-to-majority round trip plus a disk flush, so latency is at least one network round trip (about 0.5 ms in a data center, 60–150 ms across regions) and throughput is capped by one leader's log. A well-run group commits thousands to tens of thousands of small writes per second, depending on hardware and batching, but it does not scale out by adding nodes: more nodes means more votes to collect, not more capacity. Databases that need both scale and consensus run many small groups, one per shard, and keep each group's data small.

Consistency in Snip

Snip is a good model for the discipline because almost none of it needs the expensive end of the table. Go through each data path and pick the weakest model that keeps users from noticing.

Path	What a wrong read looks like	Chosen guarantee
Create a link, then open it	The creator clicks "copy" and gets 404 from a lagging replica.	Read-your-writes for the creator, by routing their reads to the shard's primary for a brief window after the write, as in chapter 4.
Redirect someone else's link	After an edit, some visitors reach the old URL for a few seconds.	Eventual, read from replicas and the cache. A few seconds of staleness is invisible, and the 5-minute edge TTL already sets a looser bound.
Click counts	The dashboard shows 10,412 and the true figure is 10,430.	Eventual and approximate. Clicks flow through a queue into counters, covered in chapter 6.
Shard map and primaries	Two servers each believe they own shard 17 and both accept writes.	Linearizable, kept in a small consensus store (etcd or similar), cached by every app server.

Count the strong guarantees: one. Only the shard map and the primary lock need linearizability, because two owners for one shard corrupt data. That is a few kilobytes changing a few times a day, which a three-node consensus group handles without effort. The 1,200 redirects per second never touch it. The link-creation path, around 12 writes per second, needs only a session guarantee, which costs a routing rule, not a protocol.

Where would a stronger model actually be worth it? Custom short codes, where two users must not both claim /sale. That is a uniqueness constraint, and it belongs on the primary of the shard that owns that code, enforced by a unique index in the same transaction as the insert. It is a linearizable operation on one row, on one machine. It does not need cross-region consensus, because the sharding scheme already gave the code a single owner. Consistency got cheap by deciding who owns the data.

Eventual consistency turns on one assumption: the work that brings copies together actually happens. Click counts rely on a queue delivering events, which is a flow-control problem. That is the next chapter: Asynchronous Work & Queues.

✓ Check yourself

Answer from memory before opening each one.

A colleague says "CAP means we have to pick two of consistency, availability and partition tolerance, and we picked CP." What is wrong with that sentence, and what does CAP actually say about a healthy cluster?

Show answer
You do not choose partition tolerance; networks split whether you plan for it or not. CAP says only that during a partition you must give up either linearizability or availability. It says nothing about a healthy cluster, where the trade is latency against consistency (PACELC). See CAP and PACELC.
A user saves a setting, the page reloads, and the old value shows. Ten seconds later it is correct. Name the guarantee that was missing and two ways to provide it without making the whole database linearizable.

Show answer
Read-your-writes. Route that user's reads to the primary for a short window after their write, or carry a version token with the session and have a replica wait until it has applied that version before answering. Both cost a routing or wait decision on one user's requests, not coordination on every read. See The spectrum of models.
A store has N = 5 replicas. You set W = 2 and R = 3. Does a read always see the latest acknowledged write? Then change one number to make the answer different and say what it costs.

Show answer
W + R = 5, which is not greater than N = 5, so the read set can miss the two nodes that took the write. Raising W to 3 gives overlap (6 > 5), at the cost that writes now wait for three replicas and fail if three are unreachable. Raising R to 4 also works but makes reads slower and less available. See Quorums.
Your team proposes routing all link creations through a 5-node Raft group so every write is linearizable. At 12 writes per second, would throughput be a problem? What else would you ask before agreeing?

Show answer
Throughput would be fine, since a group handles thousands of small writes per second. The tempting objection is wrong at this volume. Ask instead about data size (500 GB and growing in one group's log and state), latency (every create pays a majority round trip, more if nodes are in different regions), and what the guarantee buys. Sharding already gives each code one owner, so a unique index on that shard's primary is linearizable for uniqueness without a global log. See Consensus and Consistency in Snip.
A sloppy quorum accepts a write while two of its three owner replicas are down. What does the client see, and what can a later read with R = 2 return?

Show answer
The client sees success, because stand-in nodes acknowledged and hold the write with hints. A later read that queries the owners can return the old value until handoff delivers the write, since the stand-ins are outside the owner set that the overlap rule counts on. The system chose write availability over the freshness guarantee. See Quorums.

Takeaway

Copies disagree because writes travel at different speeds. A consistency model is the rule about which stale answers a reader may get, and it is separate from the C in ACID.
CAP covers only the partition case, where you give up linearizability or availability. The cost you pay every day is PACELC's latency against consistency, one round trip per coordinated read.
Stronger models forbid more and coordinate more. Session guarantees like read-your-writes cover most user-visible bugs for the price of a routing rule.
Quorums give tunable consistency with W + R > N overlap, but overlap is weaker than linearizability, and sloppy quorums trade even that for write availability.
Consensus gives one agreed order through a leader and a majority. Use it for small, critical facts such as leaders, locks and shard maps, and never for bulk data, because every write pays a majority round trip and one group does not scale out.
Snip needs one linearizable thing, the shard map and primary ownership, and it handles everything else with read-your-writes or eventual reads. Pay for strength per data path, not per system.

Eventual consistency only holds if the work that reconciles the copies gets done, and click counting does that work in the background at 100 million events a day. → Asynchronous Work & Queues