Reliability & Observability

Delivery semantics & effectively-once

Every hop in a pipeline — producer to queue, queue to consumer, consumer to warehouse — makes one of three promises about how many times a record is delivered. Interviewers use this vocabulary as a proxy for whether you actually understand distributed failure, so be precise.

True end-to-end exactly-once across independent systems is mostly a myth — you can't atomically commit "I read this Kafka offset" and "I wrote this warehouse row" across two systems that don't share a transaction. What you can build, and what senior engineers actually say, is effectively-once: at-least-once delivery plus idempotent processing plus dedup. You accept that records may arrive twice; you make the second arrival a no-op.

Idempotency keys & atomic swaps

Idempotency needs a key: a stable identifier that lets you recognize "I have already applied this." It should come from the data or the request, never from now() or a random UUID generated at processing time — those change on every retry and defeat the whole point.

-- At-least-once delivery + a natural idempotency key (order_id) + a
-- source sequence (updated_at). Re-running with duplicate rows is a no-op,
-- and stale duplicates never clobber newer state.
MERGE INTO analytics.orders AS tgt
USING (
    SELECT order_id, status, amount_cents, updated_at,
           ROW_NUMBER() OVER (
               PARTITION BY order_id ORDER BY updated_at DESC
           ) AS rn
    FROM staging.orders_incoming        -- may contain dupes from retries
) AS src
ON tgt.order_id = src.order_id AND src.rn = 1
WHEN MATCHED AND src.updated_at > tgt.updated_at THEN UPDATE SET
    status = src.status, amount_cents = src.amount_cents,
    updated_at = src.updated_at
WHEN NOT MATCHED AND src.rn = 1 THEN INSERT
    (order_id, status, amount_cents, updated_at)
    VALUES (src.order_id, src.status, src.amount_cents, src.updated_at);

For file- and partition-based systems, the reliability primitive is the atomic swap: never mutate the live table in place. Write the new partition or table to a side location, validate it, then flip a pointer in a single metadata operation. Readers see either the old version or the new version, never a half-written one. This is exactly the Write-Audit-Publish flow from Chapter 06, and it's why table formats like Iceberg and Delta (Chapter 08) matter — the commit that swaps the manifest is atomic.

The dual-write problem & the outbox

Here's the trap that separates L2 from L4. Your service updates its own database and needs to emit an event ("order placed") so the pipeline picks it up. The naive code does both:

# DUAL-WRITE BUG: two systems, no shared transaction.
def place_order(order):
    db.insert(order)              # commit #1: Postgres
    kafka.produce("orders", order)  # commit #2: Kafka
    # Crash between the two lines -> DB has the order, Kafka never
    # heard about it. Or Kafka succeeds and the DB txn rolls back ->
    # a phantom event for an order that doesn't exist. Silently wrong.

You cannot make two independent systems commit atomically. The fix is the transactional outbox: write the event into an outbox table in the same local transaction as the business change. Now there's one commit, so the state and the intent-to-publish succeed or fail together. A separate relay process (or log-based CDC — Chapter 05) reads the outbox and publishes downstream, marking rows sent.

-- 1. One local transaction writes BOTH the state change and the event.
BEGIN;
  INSERT INTO orders (order_id, status, amount_cents)
  VALUES ('ord_123', 'placed', 4200);

  INSERT INTO outbox (event_id, aggregate_id, event_type, payload, created_at)
  VALUES (gen_random_uuid(), 'ord_123', 'order.placed',
          '{"order_id":"ord_123","amount_cents":4200}', now());
COMMIT;

-- 2. A relay (or Debezium CDC on the outbox table) publishes unsent rows,
--    at-least-once, then marks them. event_id is the idempotency key so
--    consumers dedup a re-published event safely.
SELECT event_id, event_type, payload
FROM   outbox
WHERE  published_at IS NULL
ORDER  BY created_at
LIMIT  500;
-- ...publish to Kafka... then:
UPDATE outbox SET published_at = now() WHERE event_id = ANY(:sent_ids);

Retries, backoff, timeouts & DLQs

Retries are how at-least-once is actually implemented, but naive retries cause more outages than they cure. Three rules:

• Exponential backoff with jitter. Fixed-interval retries from a thousand workers create a synchronized thundering herd that hammers the recovering dependency at exactly the wrong moment. Multiply the delay each attempt and randomize it so retries spread out.
• Timeouts on everything. A call with no timeout isn't a call, it's a resource leak waiting for a hang. Every network and query call gets a bounded timeout; the timeout is a failure that triggers the retry policy.
• Bounded retries, then a dead-letter. Retrying forever on a poison message — a record that will never succeed because it's malformed or violates a constraint — blocks the whole partition behind it. Cap the attempts and route the loser to a dead-letter queue.

import random, time

def with_retries(fn, key, max_attempts=5, base=0.5, cap=30.0):
    """At-least-once processing. `key` is the idempotency key so a retry
    that actually succeeded server-side but lost the ack is harmless."""
    for attempt in range(max_attempts):
        try:
            return fn(key, timeout=10)          # bounded timeout, always
        except RetryableError as e:
            if attempt == max_attempts - 1:
                dead_letter(key, reason=str(e))  # poison msg -> DLQ, don't block
                raise
            # exponential backoff + full jitter (AWS "Exponential Backoff and Jitter")
            delay = random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(delay)
        except NonRetryableError:
            dead_letter(key, reason="permanent")  # 400-class: never retry
            raise

Distinguish retryable failures (timeouts, 5xx, throttling, a lock conflict) from non-retryable ones (schema violation, 4xx, a poison record). Retrying a permanent error just wastes the retry budget and delays the DLQ. The DLQ itself must be monitored and drainable — a DLQ nobody watches is just silent data loss with extra steps.

Failure domains, blast radius & restartability

Reliability isn't only "does one task recover" — it's "when something fails, how much goes down with it?" That's the blast radius, and containing it is a design activity, not a monitoring one.

• Failure domains. A shared upstream (one credentials secret, one Spark cluster, one schema-registry) is a domain: when it fails, everything behind it fails together. Draw them explicitly. The goal is that a bad partition for one source doesn't halt the other twenty.
• Isolate the poison. Per-source DAGs or per-table tasks (see task-centric vs asset-centric in Chapter 03) mean one broken feed quarantines to its own lane instead of failing a monolithic run.
• Graceful degradation. If the enrichment API is down, can you serve yesterday's dimension and flag the rows stale, rather than failing the whole load? Partial-but-labeled often beats all-or-nothing.
• Checkpointing & restartability. A job that must restart from zero after a failure at 90% doesn't scale. Checkpoint progress (processed offsets, last completed partition, a high-water mark) so a restart resumes, not repeats. Idempotency makes the overlap on resume safe.

     FAILURE DOMAIN A                 FAILURE DOMAIN B
  ┌──────────────────────┐        ┌──────────────────────┐
  │ payments CDC ─┐       │        │ app events (Kafka) ─┐ │
  │ ledger  CDC ──┼─►stage│  ⚡    │ clickstream ────────┼─┤
  └───────────────┼──────┘        └─────────────────────┼─┘
                  ▼                                      ▼
            ┌───────────── shared transform (Spark) ─────────────┐  ⚡ shared
            │  checkpoint every partition · at-least-once + dedup │     domain!
            └───────────────────────┬────────────────────────────┘
                                     ▼   atomic swap (WAP)
                              ┌────────────┐
                              │ marts (served)│  freshness SLA: 2h
                              └────────────┘
  ⚡ = likely failure point.  If the Spark cluster is the only compute,
      domains A and B collapse into one blast radius — split or add fallback.

The observability pillars

Application observability is metrics/logs/traces. Data observability adds a second axis: the pipeline can be green while the data is wrong. A job that "succeeds" but writes zero rows, or yesterday's rows, or rows with a shifted schema, is the failure mode that actually pages you. Watch five pillars:

Freshness and volume are the highest-ROI pair — they're cheap to compute and catch the majority of real incidents (a stuck upstream, a silently empty load). Distribution monitoring is closer to the anomaly detection in Chapter 06; you can buy it (Monte Carlo, Metaplane, Soda) or roll your own with control charts on the metrics above. The point in an interview: name the pillars, then say which you'd instrument first and why.

Lineage

When a metric is wrong at 9am, lineage is what turns "somewhere upstream" into "this column, from this source, changed at 2am." Two granularities:

• Table-level lineage — which tables feed which. dbt gives you this for free from ref()/source(); the DAG is the lineage graph.
• Column-level lineage — which source columns compute which output columns. This is what lets you scope a schema change or a restatement precisely instead of re-running everything downstream.

OpenLineage (with Marquez as a reference backend) is the emerging open standard for emitting lineage events from Airflow, Spark, and dbt in a tool-neutral way — worth naming as the interoperable answer. On the consumer side, dbt exposures declare the dashboards, ML models, and apps that depend on your marts, so lineage extends past the warehouse to "who actually breaks if this table is late." That last mile is what converts a data incident into a stakeholder communication.

SLAs, SLOs & error budgets

These three get muddled constantly; keeping them straight is itself a signal. For data:

• SLI — the measured indicator. "Data freshness = minutes between event time and availability in the mart."
• SLO — your internal target on that SLI. "Freshness < 2h for 99% of days." This is what you engineer and alert against.
• SLA — the externally-promised, often contractual version, usually looser than the SLO, with consequences if breached. You set the SLO tighter so you notice before the SLA is at risk.

An error budget falls straight out of the SLO: a 99% freshness target means 1% of the period is your budget to spend on failures, maintenance, and risky changes. Budget intact ⇒ ship faster; budget exhausted ⇒ freeze changes and stabilize. It turns "how reliable is reliable enough?" from an argument into arithmetic.

-- A real freshness SLI, computed on the DATA, not on job status.
-- Run it as a monitor; page if it breaches the SLO.
SELECT
    max(event_time)                                   AS last_event,
    datediff('minute', max(event_time), current_timestamp) AS staleness_min,
    120                                               AS slo_minutes,
    datediff('minute', max(event_time), current_timestamp) > 120 AS slo_breached
FROM analytics.orders;

Alerting, on-call & incident response

Good instrumentation with bad alerting is worse than no alerting — a flood of noisy pages trains everyone to ignore the one that matters. Design alerts to be:

• Actionable. Every page maps to a decision. "CPU 80%" isn't actionable; "orders freshness breached SLO, 0 rows in 3h" is.
• Deduped & grouped. One upstream outage that fails forty tables is one incident, not forty pages. Group by root cause using lineage.
• Routed & tiered. Page for SLO breaches on tier-1 data; send a ticket or Slack for warnings. Match severity to consumer impact — the exposures from the lineage section tell you the tier.
• Warn vs block. Same decision as Chapter 06: a failed critical check should halt promotion (circuit breaker), while a soft anomaly warns. Blocking bad data beats alerting on it after it's live.

When something does break, run a repeatable loop rather than improvising:

DETECT   → monitor/SLO breach fires (not a consumer email). Ack, declare.
CONTAIN  → stop the bleeding: pause the DAG, freeze promotion (WAP holds
           bad data in staging), flag affected marts as "stale/suspect".
DIAGNOSE → use lineage to find the changed source/column + the blast radius
           (which downstream exposures are impacted).
REMEDIATE→ backfill or RESTATE the affected partitions idempotently
           (immutable partitions + re-run, per Ch04/05). Reconcile after.
PREVENT  → add the test/contract/monitor that would have caught it earlier;
           write the blameless postmortem with an action item + owner.

The runbook is what makes this executable at 3am by whoever is on-call, not just the author — the alert links to the runbook, the runbook has the exact queries and the rollback command. The blameless postmortem is where reliability compounds: each incident should end with a durable control (a new freshness monitor, a data contract, a reconciliation tolerance) so the same class of failure can't recur silently.

Leveling ladder

Reliability is where the ladder is steepest — the same scenario ("make this pipeline dependable") is answerable at every rung, and interviewers climb it deliberately.

Next, the other half of production-grade: making all of this hold up as data and traffic grow by orders of magnitude — 08 · Scale & Architecture.

✓ Check yourself

Answer from memory before opening each one.

1. A consumer reads from Kafka and writes to a warehouse. Why can you not make that pair exactly-once, and what do you build instead?

   Show answer

   The consumer offset and the warehouse write live in two systems with no shared transaction, so a crash between them either repeats or loses a record. Build effectively-once: at-least-once delivery, an idempotent write keyed on the record, and a dedup rule, so the second arrival is a no-op. Exactly-once is real only inside one closed system. See delivery semantics.

2. A load dedups on a UUID generated when the batch is processed. A retry reprocesses the same records. What is in the table, and what should the key be?

   Show answer

   Every retried record carries a new UUID, so every one is inserted again and the dedup does nothing. The key must come from the data or the request, such as an order id with a source version, so the same record maps to the same key on every attempt. A now() value fails the same way. See idempotency keys.

3. A service inserts an order and then produces a Kafka event. The process dies between the two calls. What does each side hold, and how does an outbox change the failure?

   Show answer

   The database has the order and Kafka never heard of it; the reverse order of calls would give a phantom event. With an outbox the order and the event row commit in one local transaction, and a relay or CDC publishes from the table later. The failure becomes a delayed or repeated publish, which a consumer handles with the event id as the idempotency key. See the dual-write problem.

4. A thousand workers retry a failing API every five seconds, and one malformed record fails every attempt. What do you change for each problem?

   Show answer

   For the herd, use exponential backoff with jitter so retries spread out instead of hitting the recovering service together. For the record, classify it as non-retryable or cap the attempts and send it to a dead-letter queue, which you monitor and can drain, so it stops blocking the partition behind it. Retries also require idempotent operations. See retries, backoff, timeouts and DLQs.

5. The Airflow scheduler is down overnight and no run starts. Task-level SLA callbacks stay silent. What does the monitor look like that does catch it, and what does a 99 percent freshness SLO give you?

   Show answer

   Compute freshness on the served data, as the gap between now and the latest event time against the target, independent of job status. A task SLA only fires for a task that ran, so a DAG that never triggered goes unreported. The SLO leaves a small error budget; while it is intact you ship, and once it is spent you freeze risky changes and stabilise. See SLAs, SLOs and error budgets.

6. Two source groups feed one Spark cluster, then one set of marts. A bad file from one source group fails the run. How do you limit what goes down, and what do you give consumers meanwhile?

   Show answer

   Split by source into separate runs or tasks so one broken feed stays in its own lane, and note that one shared cluster still joins the domains, so add fallback capacity or split it. Meanwhile serve the last good data and flag affected rows stale instead of failing the whole load, and checkpoint so a restart resumes rather than repeats. See failure domains and blast radius.

Takeaway

• End-to-end exactly-once across independent systems is not available; build at-least-once delivery with idempotent processing and call it effectively-once.
• An idempotency key comes from the data, never from the clock or a random value made at processing time.
• Writes are swaps, not in-place updates: stage, validate, then publish with one atomic metadata flip.
• A service that writes to its database and a broker has a dual-write bug; the outbox turns it into one local commit plus an idempotent publish.
• Retries need backoff with jitter, timeouts, a retry cap, a monitored dead-letter queue, and idempotent operations underneath.
• Measure freshness and volume on the data, because a green job can write nothing; task SLAs do not see a run that never started.
• Contain blast radius by splitting failure domains, serving labeled stale data, and checkpointing so restarts resume.
• Run incidents as detect, contain, diagnose, remediate and prevent, and end each with a durable control.

Reliability holds only if the design also holds as data and traffic grow, which is the question of scale. → Scale & Architecture

Further reading

• Service Level Objectives — Google SRE Book, 2016. SLIs, SLOs and SLAs, and how to choose targets.
• Embracing Risk — Google SRE Book, 2016. Where error budgets come from and how they settle release arguments.
• Postmortem Culture: Learning from Failure — Google SRE Book, 2016. Blameless postmortems and what makes follow-up actions stick.
• Timeouts, retries and backoff with jitter — Amazon Builders' Library. Why naive retries amplify outages, and how jitter and timeouts fix it.
• Debezium: Outbox Event Router — Debezium docs. The transactional outbox pattern implemented with change data capture.
• OpenLineage documentation — OpenLineage project. The open standard for emitting lineage events from Airflow, Spark and dbt.
• Idempotence Is Not a Medical Condition — Pat Helland, ACM Queue, 2012. Why at-least-once messaging forces idempotent processing.