Distributed Systems II — Consistency & Delivery
Once data lives on many machines connected by a network that can drop and delay messages, a hard question appears: how do the machines agree on what's true? This is the chapter behind "two services disagree" and "we're seeing duplicate events." It closes Foundations with the ideas that protect your pipelines from a network that refuses to be reliable — and gives you idempotency, the single most useful trick in the whole course.
The core problem: agreement over a flaky network
In Chapter 10 we scaled out — splitting work across many machines and copying (replicating) the same data to several nodes so reads are fast and a dead machine doesn't lose your data. "Just copy the data everywhere" sounds simple, and on a good day it is. The trouble is the network in between.
A network is not a reliable wire. Messages get dropped, delayed, reordered, and sometimes a whole group of machines gets cut off from another group — a network partition. Crucially, when a node sends a message and hears nothing back, it cannot tell whether the other node crashed, is just slow, or the reply got lost. The silence looks identical in all three cases.
That single picture is the whole chapter. The moment a partition happens, the copies can disagree, and each node has to decide — alone — how to behave. Everything that follows (CAP, consistency models, consensus, delivery, idempotency) is a different angle on this one problem: how do independent machines stay in agreement when the network won't cooperate?
You won't write a database from scratch. But you will choose and operate systems that made these trade-offs for you — and when a dashboard shows a stale number or a queue delivers the same event twice, you'll need to know which trade-off you're standing on. These ideas turn "the system is being weird" into "ah, that's eventual consistency / an at-least-once retry."
The CAP theorem, in plain language
The CAP theorem is the most famous (and most misquoted) result in distributed systems. The three letters stand for:
- C — Consistency: every read sees the most recent write. All nodes show the same value at the same time.
- A — Availability: every request gets an answer (not an error or a hang), even if some nodes are unreachable.
- P — Partition tolerance: the system keeps working even when the network splits into groups that can't talk.
The popular phrasing — "pick 2 of 3" — is misleading. Across real networks, partitions will happen; P is not optional. So the honest reading is narrower and far more useful:
When a partition is happening, you must choose: stay Consistent (refuse some requests) or stay Available (answer, but maybe with stale or conflicting data). You cannot have both during the partition. When the network is healthy, you get both — CAP only forces the choice during the split.
| System / situation | Leans toward | Why that's the right call |
|---|---|---|
| Bank account balance, money transfer | C (CP) | Showing a wrong balance or double-spending is unacceptable — better to reject the request and retry than to lie. |
| Shopping cart, social feed, "likes" | A (AP) | A briefly stale cart or like-count is fine; a site that won't load loses money. Reconcile later. |
| Leader-based SQL database (e.g. Postgres primary) | C | Writes go to one leader; if it's unreachable, writes pause rather than diverge. |
| Multi-region key-value store (e.g. Dynamo-style) | A | Built to keep serving from any region during a split, accepting that copies converge afterward. |
There's no universally "right" answer — only the right answer for this data. Part of reading a system's docs is spotting which way it leans, because that decision shows up in your data on the worst day.
Strong vs eventual consistency
"Consistency" isn't all-or-nothing; it's a spectrum of guarantees a system makes about what a read can see. Two anchor points matter most to a data engineer:
| Strong consistency | Eventual consistency | |
|---|---|---|
| The promise | A read always returns the latest write. Once you write x = 9, everyone sees 9. | If writes stop, all copies eventually agree — but for a while, a read may return an older value. |
| What you might see | Always the current truth. | Slightly stale data — last second's value, not this second's. |
| Cost | Coordination on every write (slower, can stall during partitions). | Cheap, fast, stays available — at the price of temporary disagreement. |
| Where you'll meet it | The primary of your transactional database; a leader-elected store. | A read replica; a CDN; a cross-region cache; many cloud object stores. |
The phrase that trips people up is "eventually". It does not mean "broken" — it means "correct, after a short catch-up window." A classic example: you write a row to your database's primary, then immediately read it back from a read replica. The replica hasn't received the change yet, so your read returns the old row. Nothing is malfunctioning; the replica is eventually consistent and you read it a half-second too soon. This is the mechanism behind "I just updated it but the dashboard still shows the old number."
The practical takeaway for a DE: always know whether you're reading a primary (strong) or a replica (eventual). Pointing a "did the order go through?" check at a read replica will give you flaky, intermittently-wrong answers — and you'll waste a day blaming your code. Pointing analytics at a replica, on the other hand, is exactly right: you don't need this-millisecond freshness, and you take load off the primary.
Consensus: how a cluster agrees on one truth
If nodes can disagree, how does a cluster ever decide a single value — like "node 3 is now the leader," or "this is record #847 in the log"? That's the consensus problem: getting a group of machines to agree on one thing despite some of them being slow, crashed, or briefly unreachable. The famous algorithms are Paxos and Raft. We'll skip the math entirely and keep just the intuition, because that's all you need to operate these systems.
Two ideas carry the whole thing:
- A majority must agree (a quorum). A value is only "real" once more than half the nodes have accepted it. With 5 nodes you need 3; the cluster survives losing 2. The magic is that any two majorities of the same group always share at least one node — so they can't commit two conflicting values. That overlap is what prevents a "split brain" where each side thinks it won.
- A leader coordinates. Rather than everyone proposing at once (chaos), the cluster elects one leader to order writes. If the leader vanishes, the survivors hold an election and a majority picks a new one. Writes pause for that brief election, then resume — this is the system choosing consistency over momentary availability.
| You'll find consensus inside… | Doing what |
|---|---|
| Distributed databases (CockroachDB, Spanner, etcd, ZooKeeper) | Agreeing on each committed write and on cluster membership. |
| Kafka (its controller / metadata quorum) | Electing the controller and agreeing which broker leads each partition. |
| Schedulers & orchestrators (Kubernetes, Nomad) | Electing one active leader so two schedulers don't fight over the same work. |
You rarely call a consensus algorithm directly. But "needs a majority to make progress" explains a lot of real behavior — like why a 3-node cluster goes read-only when 2 nodes die, or why people run clusters with an odd number of nodes (so a majority is always well-defined).
Delivery semantics — the practical heart
Now the part you'll touch every single week. When one system sends a message to another over a flaky network — a producer to a queue, a queue to your pipeline — what guarantee do you get that the message arrives exactly as intended? There are three possible promises, and the differences are the source of a huge share of real pipeline bugs.
| Semantic | Guarantee | Failure mode | Use it when… |
|---|---|---|---|
| At-most-once | Delivered zero or one time. | Can silently lose data. | Loss is acceptable — metrics, sampled logs, best-effort telemetry. |
| At-least-once | Delivered one or more times — never lost. | Can duplicate data. | Almost everything — the practical default for queues and pipelines. |
| Exactly-once | Effectively one time, no loss, no dups. | Complex, slower, and only within a system's boundaries. | You truly need it and can pay for it (financial ledgers, some stream processors). |
Why does at-least-once duplicate? Because of that same unreliable network and the silence problem from the start of the chapter. The sender delivers a message and waits for an acknowledgement ("got it"). If the work succeeded but the ack got lost on the way back, the sender hears nothing, assumes failure, and retries — delivering the same message a second time.
True end-to-end exactly-once delivery across independent systems is, in the general case, impossible — the ack problem above can't be wished away. What systems advertise as "exactly-once" is usually at-least-once delivery plus deduplication, working only inside their own boundary. The robust, portable answer isn't to chase magical delivery; it's to make your processing safe to repeat. That's idempotency, next — and it's how real pipelines achieve effectively exactly-once.
Idempotency — the superpower that tames at-least-once
Here's the key insight that makes peace with a duplicating world: if running an operation twice has the same effect as running it once, then duplicates stop mattering. An operation with that property is called idempotent. You met this in Course 2 when you built a pipeline you could safely re-run after a crash without doubling your data — that re-runnability was idempotency. Now you know the name and the why.
Compare two ways to load a payment event:
| Not idempotent (dangerous) | Idempotent (safe to retry) |
|---|---|
balance = balance + 100Run twice → balance grows by 200. A duplicate corrupts the data. | INSERT … ON CONFLICT (event_id) DO NOTHINGRun twice → the second insert is a no-op. Same result either way. |
| "Append this row" (blind insert) | "Set this row to this value" (upsert keyed by id) |
The three workhorse techniques — you'll reach for these constantly:
- Idempotency key. Give every event a stable unique id (an
event_id, an order number, a hash of the payload). Carry it through the pipeline so a retry is recognizable as the same event, not a new one. - Upsert instead of insert. "Insert-or-update by key" (
MERGE/ON CONFLICT) makes a write land in the same final state no matter how many times it runs. - Dedup on the key. Keep a record of ids you've already processed (a table, a set, a unique constraint) and drop anything you've seen before.
Stop trying to guarantee a message arrives exactly once — you usually can't. Instead, assume duplicates will happen and make them harmless. Design every load step so re-running it is a non-event. Do that, and a retry-happy queue, a crashed-and-restarted job, and a replayed backfill all become safe by construction. This one habit prevents more data-quality incidents than almost anything else in the craft.
Wrapping up Foundations
Take a breath — you just finished the longest, deepest course in the curriculum. Look at the distance you've covered. You can now narrate the entire journey of data, end to end:
When you write SELECT … JOIN … GROUP BY now, you don't see magic — you see bytes moving from disk into memory, hashed and sorted by real algorithms (Ch. 2–4), maybe spilling when memory runs out (Ch. 8), possibly fanned across a cluster that had to agree on the result (Ch. 10–11). That narration — knowing what physically happens — is exactly the senior engineer's superpower the first chapter promised you'd build. It's the thing that turns "the system is being weird" into a specific, answerable question.
Foundations told you how the machines work. Course 4 puts it to work. You'll build real pipelines — ingestion, transformation with dbt, orchestration, testing, and the day-to-day craft of moving data reliably — and every chapter will lean on what you just learned. That "design it to be safe to retry" instinct from idempotency? You'll use it on day one. Foundations was the theory; the Craft is where it earns its keep.
If any chapter here felt shaky, that's expected — Foundations is a reference you return to, not a wall you clear once. Distributed systems especially will click harder after you've built a pipeline that hits a duplicate event for real. Coming back is part of the plan.
✓ Check yourself
- During a network partition, what does a system give up if it chooses Consistency? What if it chooses Availability?
- Why might reading from a read replica return stale data — and is that a bug?
- In one sentence: why does at-least-once delivery cause duplicates?
- What does it mean for an operation to be idempotent, and why does that make duplicates harmless?
Exercise — Make a duplicating load idempotent
Your pipeline reads order events from an at-least-once queue and writes them to a table. Because the queue sometimes redelivers, you occasionally process the same order_id twice and end up with duplicate rows (and, worse, a downstream revenue total that's too high). Describe two different ways to make the load idempotent so duplicates become harmless.
Try to write both before expanding.
-- WAY 1: Upsert keyed by the event's unique id.
-- A redelivery overwrites the same row instead of adding a new one,
-- so the final state is identical no matter how many times it runs.
INSERT INTO orders (order_id, customer, amount, ordered_at)
VALUES (:order_id, :customer, :amount, :ordered_at)
ON CONFLICT (order_id) DO UPDATE
SET customer = EXCLUDED.customer,
amount = EXCLUDED.amount,
ordered_at = EXCLUDED.ordered_at;
-- (Requires a UNIQUE / PRIMARY KEY on order_id.)
-- WAY 2: Dedup — record processed ids and skip ones you've seen.
-- The blind insert simply does nothing on a repeat.
INSERT INTO orders (order_id, customer, amount, ordered_at)
VALUES (:order_id, :customer, :amount, :ordered_at)
ON CONFLICT (order_id) DO NOTHING;
-- Equivalently: keep a `processed_ids` table (or set) and check membership
-- before doing the work, inserting the id in the same transaction.Other valid answers: a MERGE statement (same idea as the upsert); deduping in a staging step before the final load; or, if the data is naturally a "current state," DELETE-then-insert keyed by id within one transaction. The common thread is always the same — a stable id plus a write that lands in the same final state every time. That's idempotency, and it's the standard way real pipelines turn an at-least-once source into an effectively exactly-once result.
Next
That's it — you've finished the longest and most demanding course in the curriculum. Bits to agreement, you can now narrate the whole journey of data through a machine and across a cluster. Be a little proud of that; most people who use these systems never learn how they actually work.
From here, the theory goes to work. Course 4 — The DE Craft is where you build real pipelines on top of everything you just learned — and it's ready now. That's your next stop.