Data Quality, Observability & Reliability

Trust is the product

Picture the Monday exec review. The GMV slide shows a number that is 12% lower than last week, and someone built a narrative around it before anyone notices the rental-meter feed silently stalled on Saturday. The number was wrong, the decision discussed off it was wrong — and worse, the next time the dashboard is right, no one believes it. That is the asymmetry that makes data quality a first-class system: one wrong number in an exec dashboard discredits the entire platform, including all the numbers that were correct.

The cost of bad data compounds across three channels. Decisions made on wrong numbers are wrong, and they are made by the most senior, most expensive people in the building. Money moves on billing and payout data — a double-counted rental_meter row (recall chapter 01) is a real overcharge and a chargeback. ML trains silently on corrupted features: a reliability model that ingested a week of telemetry gaps as genuine downtime will mis-rank healthy hosts for months. The decision error is loud; the ML error is silent and slow, which makes it the more dangerous of the two.

The dimensions of data quality

"Is the data good?" is unanswerable until you decompose good into measurable dimensions. Each dimension catches a distinct failure class, and a mature platform monitors all seven. Map them to the case study so they stay concrete.

Testing: asserting what you know

Tests encode known expectations as code that runs in the pipeline and fails loudly when violated. They are the foundation we introduced in chapter 07 on transformation; here we organize them into a deliberate pyramid.

The base of the pyramid is the four dbt built-in generic tests — declared in YAML next to the model and run on every build. They cost almost nothing and catch the bulk of mechanical breakage.

models:
  - name: fct_rental_meter
    description: "One row per rental-hour. The billing atom (ch. 06)."
    columns:
      - name: meter_id
        tests:
          - not_null
          - unique
      - name: invoice_id
        tests:
          - not_null
          - relationships:           # referential integrity
              to: ref('dim_invoice')
              field: invoice_id
      - name: meter_status
        tests:
          - accepted_values:
              values: ['open', 'billed', 'voided', 'credited']
      - name: metered_hours
        tests:
          - dbt_utils.accepted_range:   # from the dbt_utils package
              min_value: 0
              max_value: 1     # a rental-hour row can't exceed one wall-clock hour

One rung up, custom and dbt_utils tests express business invariants the generics can't. A custom singular test is just a SQL query that should return zero rows; if it returns any, the test fails.

-- A (rental_id, billing_hour) must appear exactly once.
-- A duplicate means a telemetry/CDC replay double-billed the customer.
select
    rental_id,
    date_trunc('hour', meter_start_ts) as billing_hour,
    count(*) as n
from {{ ref('fct_rental_meter') }}
group by 1, 2
having count(*) > 1      -- any returned row = test failure

For the richer, suite-style checks — null-rate thresholds, distribution drift, freshness — teams reach for Great Expectations or Soda, which sit alongside dbt and express expectations declaratively, often against sources dbt doesn't transform (the raw telemetry landing zone, for instance).

checks for raw_telemetry_sample:
  - freshness(event_time) < 5m            # firehose should never lag 5 min
  - row_count between 130000000 and 175000000   # ~150M/day expected band
  - missing_percent(instance_id) < 60%    # idle GPUs are null; >60% is suspect
  - invalid_percent(gpu_util_pct) = 0:
      valid min: 0
      valid max: 100                       # util is a percentage; 4000 is impossible
  - duplicate_count(machine_id, gpu_index, event_time) = 0   # grain must be unique
  - schema:
      fail:
        when required column missing: [machine_id, gpu_index, event_time, gpu_util_pct]
        when wrong column type:
          event_time: timestamp

Finally, contract tests (from chapter 01) run at the boundary — in the producing team's CI, before a schema change ships. They are the cheapest tests of all because they fail in a pull request rather than at 3 a.m. in production. The contract's declared schema, required, and unique fields become assertions the producer cannot break without bumping the major version and giving notice.

Data observability: catching what you didn't anticipate

Testing scales linearly with foresight — you only catch the failures you imagined and wrote SQL for. At GridDP's surface area (dozens of sources, hundreds of models), that coverage is always incomplete. Data observability inverts the model: instead of you declaring expectations, the system learns the normal behavior of each table — its typical freshness, row volume, schema, and column distributions — and alerts on statistically significant deviations. This is the category occupied by tools like Monte Carlo and Anomalo.

The power is catching unknown-unknowns. No one wrote a test asserting "the null rate on instance_id in fct_rental_meter stays near 8%," but a monitor that learned the baseline fires the moment it jumps to 40% — surfacing a join bug the day it ships rather than the day finance notices revenue looks off. Observability monitors fall into two layers:

• Table monitors — freshness, volume, schema, and per-column distribution anomalies, learned automatically across every table the platform registers. Coverage is broad and requires almost no per-table authoring.
• Pipeline monitors — instrument the machinery, not just the output: Kafka consumer lag on the telemetry topic, CDC replication slot lag off the Postgres WAL, dbt run duration and row deltas per model, Dagster/Airflow task failures and retries. A freshness problem is usually a symptom; the pipeline monitor names the cause.

Column-level lineage & impact analysis

When a test or monitor fires, the first two questions are always: what broke upstream to cause this, and what downstream is now poisoned. Both are answered by lineage — the dependency graph from source columns through transformations to the dashboards and ML features that consume them. Table-level lineage tells you which models depend on a table; column-level lineage tells you that the GMV metric depends specifically on fct_rental_meter.amount_usd, so a problem in an unrelated column doesn't spuriously page the GMV owner.

This graph is what makes triage fast. The instant the amount_usd monitor fires on stg_rental_meter, impact analysis enumerates the blast radius — the GMV metric, the eight dashboards built on it, the finance close, and the dynamic-pricing feature (downstream and into ch. 12) — so on-call can quarantine the affected marts and proactively message the eight dashboard owners before they message us. Run in reverse, the same graph does root-cause analysis: walk upstream from the broken metric until you reach the first table where the anomaly appears, and that's your culprit. Lineage is harvested automatically from dbt's compiled DAG and the warehouse query logs, then surfaced in the catalog (DataHub / Unity Catalog / Horizon, per the Start Here stacks).

Data SLAs & SLOs: quality as a contract

Not every dataset deserves the same rigor, and pretending otherwise either bankrupts the team (everything is tier-1) or burns trust (nothing is). The answer is to set SLOs per dataset tier — reusing exactly the tier model from chapter 09. An SLA is the promise made to consumers ("GMV is fresh within 15 minutes, 99.9% of the time"); the SLO is the internal target the platform engineers against; and the error budget is the allowed shortfall (0.1% of the time it may breach) that governs how hard you fight every minor blip.

Data incident response

Quality fails eventually; what separates a mature platform is the response loop. Data incidents borrow the SRE lifecycle, adapted for the fact that the "outage" is a wrong number rather than a down service — which means the hardest stage is often detection, because wrong data looks exactly like right data until someone reconciles it.

• Detect. Best case, a monitor or test fires before a human notices. Worst case — and it happens — detection is the "this dashboard looks wrong" Slack message from an analyst. Treat that message as a sev signal, not an annoyance; it means your monitoring had a gap.
• Triage. Use lineage (§lineage) to fix severity by the highest tier in the blast radius and to decide who owns the fix. A tier-1 mart in the radius is an immediate page.
• Communicate. The single most trust-preserving action is telling people the data is wrong before they act on it. A pinned message in #data-alerts — "GMV dashboard is stale as of 09:14, root cause being investigated, do not trust until cleared, ETA 40m" — is worth more than a silent fast fix.
• Mitigate. Two standard moves: quarantine (flag the affected marts unhealthy / hide them in the BI layer so no one reads bad data) and rollback (revert the table to its last known-good run, e.g. via the table format's time-travel or by re-running from the last good partition), then backfill once the upstream fix lands.
• RCA. A blameless post-mortem whose mandatory output is a new test or monitor. An incident that doesn't tighten the safety net is an incident you'll have again. This is the feedback loop that makes the whole system get stronger over time.

This implies on-call for data — a rotation, a sev scale, and runbooks per tier-1 dataset ("if the ledger reconciliation fails: here's the query to find the imbalanced entries, here's how to quarantine, here's who in finance to notify"). Data on-call is not yet universal, but at a company where data sits in the product's critical path, it is non-negotiable.

Worked example: guarding the billing domain

The billing domain (chapter 01) has the highest correctness bar in the company, so it is where the whole system earns its keep. We layer three defenses on it, one per relevant dimension.

1 · A freshness monitor on rental_meter — a tier-1 SLO of < 15 min. If the CDC feed off the billing Postgres WAL stalls, this fires and pages on-call before the stale number reaches the Monday exec review. This is the Saturday-stall scenario from §why, now caught.

2 · A hard assertion that the double-entry ledger balances. The invariant from chapter 01 is simple and absolute: every transaction's debits equal its credits, so the books must net to zero. This is a custom dbt test — a query that must return zero rows — and it blocks the pipeline, because a non-balancing ledger must never reach a dashboard.

-- INVARIANT (ch. 01): in double-entry accounting, every transaction's
-- debits must equal its credits. Any transaction that nets non-zero is
-- a corruption that must NEVER reach a financial dashboard.
-- This test returns the offending transactions; zero rows = pass.
with per_txn as (
    select
        transaction_id,
        round(sum(case when entry_type = 'debit'  then amount_usd else 0 end), 4) as debits,
        round(sum(case when entry_type = 'credit' then amount_usd else 0 end), 4) as credits
    from {{ ref('fct_ledger_entry') }}
    group by transaction_id
)
select
    transaction_id,
    debits,
    credits,
    debits - credits as imbalance
from per_txn
where debits <> credits        -- any row here fails the build

3 · Reconciliation to Stripe. The ledger balancing proves the books are internally consistent; it does not prove they match the money that actually moved. Accuracy demands a second source of truth. A daily reconciliation ties warehouse-side settled revenue to Stripe's own settlement report, flagging any day where the two diverge beyond a rounding tolerance — catching missed webhooks, the webhook/API gap from chapter 01, and silent integration drift.

-- Warehouse-computed settled revenue must tie to Stripe's settlement
-- report to within a small tolerance (FX/rounding). Divergence = a real
-- accuracy break (missed webhook, dropped meter row, double charge).
with wh as (
    select settle_date, sum(amount_usd) as wh_revenue
    from {{ ref('fct_invoice_settled') }}
    group by settle_date
),
stripe as (
    select settle_date, sum(amount_usd) as stripe_revenue
    from {{ ref('stg_stripe_settlement') }}
    group by settle_date
)
select
    coalesce(wh.settle_date, stripe.settle_date) as settle_date,
    wh.wh_revenue,
    stripe.stripe_revenue,
    abs(coalesce(wh.wh_revenue, 0) - coalesce(stripe.stripe_revenue, 0)) as gap
from wh
full outer join stripe using (settle_date)
where abs(coalesce(wh.wh_revenue, 0) - coalesce(stripe.stripe_revenue, 0)) > 1.00

Takeaway

• Trust is the product. One wrong number in an exec dashboard discredits every correct number; bad data costs decisions, money, and silently-corrupted ML.
• Decompose "good" into seven dimensions — freshness, volume, schema, distribution/values, uniqueness, referential integrity, accuracy. The first six are checkable from inside the warehouse; accuracy needs a second source of truth and is the one finance feels.
• Tests assert the known (dbt generics → custom SQL → contract tests) and block the pipeline; observability learns the normal and catches the unknown-unknowns. Run both.
• Column-level lineage turns "something broke" into a bounded blast radius and a fast root cause; it is the connective tissue of triage.
• Set freshness/completeness/accuracy SLOs per dataset tier (ch. 09), give each an error budget, and let the tier decide who gets paged.
• Operate quality with a real incident loop — detect, triage, communicate, mitigate, RCA — whose every post-mortem ships a new test or monitor.
• The billing domain is where it all pays off: a freshness monitor, a ledger-balance assertion, and Stripe reconciliation are the three queries finance's trust hinges on.

Quality earns trust in the numbers; the next chapter earns trust in who may see them and prove it — access control, PII, lineage for compliance, and audit. → Governance & Security