Data Quality & Trust

Trust is the product

Restating chapter 01 because it governs this whole topic: a dashboard nobody believes is worse than none — it generates re-checking, shadow spreadsheets, and lost credibility. Your early roadmap should optimize for trust in a few metrics, not coverage. Every quality mechanism below exists to protect that trust.

The dimensions of data quality

Have a vocabulary for what "quality" means — interviewers want structured thinking, not "I'd add some tests."

Where to test — at the boundaries

Test where data enters and where it's consumed, not everywhere (that's noise).

• Source / ingest: schema and freshness — catch upstream changes immediately.
• Staging: uniqueness of the dedup key, not-null on keys, valid enums.
• Marts: grain uniqueness, referential integrity to dimensions, business invariants (e.g. revenue ≥ 0), reconciliation against source-of-truth.

The shift-left principle: catch problems as early in the pipeline as possible — a bad row caught at staging is cheaper than a wrong number caught in a board deck.

Test types (and the dbt vocabulary)

dbt is the lingua franca; know the four built-in generic tests and what a custom test is.

• Generic: unique, not_null, accepted_values, relationships (foreign-key). Declared in YAML, run on every build.
• Singular: a SQL query that should return zero rows; any rows = failure. This is how you encode business rules.
• Packages: dbt_utils / dbt_expectations add row-count ranges, recency, distribution, mutually-exclusive-ranges, etc.
• Severity: error blocks the build; warn surfaces without stopping. Use warn for noisy/soft checks so alerts stay meaningful.

models:
  - name: fct_usage_daily
    columns:
      - name: usage_id
        tests: [unique, not_null]
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('dim_customer')
              field: customer_id
      - name: status
        tests:
          - accepted_values: { values: ['active','stopped','failed'] }

-- Fails the build if any row violates the rule
SELECT * FROM {{ ref('fct_usage_daily') }}
WHERE gpu_seconds < 0;

Freshness & volume — the two that catch silent breaks

Most "silent" data incidents are a feed that stopped or halved. These two checks catch the majority of real-world breakage for almost no effort:

-- Freshness: alert if newest data is too old
SELECT 'usage_events stale' AS issue
WHERE (SELECT MAX(loaded_at) FROM usage_events) < NOW() - INTERVAL '2 hours';

-- Volume: alert if today is wildly off the trailing average
WITH daily AS (
  SELECT DATE(started_at) d, COUNT(*) n FROM usage_events GROUP BY 1
)
SELECT * FROM daily
WHERE d = CURRENT_DATE
  AND n < 0.5 * (SELECT AVG(n) FROM daily WHERE d >= CURRENT_DATE - 14);

dbt has native source freshness checks; mention them. As a first hire, freshness + volume + key-uniqueness on your handful of trusted metrics is 80% of the value for 20% of the effort.

Data contracts

An agreement with upstream producers about schema, types, semantics, and SLAs — so a backend deploy doesn't silently break your pipeline. The concept matters more than any tool.

• What's in a contract: column names & types, nullability, allowed values, freshness guarantee, and a change/deprecation process.
• Enforcement: schema tests at ingest that fail loudly; dbt's contract config that enforces declared column types on a model's output.
• First-hire reality: you often can't impose formal contracts on a small eng team early. The pragmatic version is a schema test that pings you the moment a column changes, plus a lightweight agreement: "tell me before you rename a field." Show you understand both the ideal and the pragmatic.

Anomaly detection

Fixed-threshold tests miss gradual drift and don't adapt to growth. Anomaly detection flags statistically unusual values:

• Simple & effective: compare today's metric to a trailing window using z-score or percent-change bands. Catches "revenue dropped 40% overnight" without hard-coding a number.
• Seasonality: weekday/weekend and month-boundary effects — compare like-for-like (this Monday vs prior Mondays) or you'll cry wolf.
• Tooling: Elementary (dbt-native, cheap to start), Monte Carlo / Bigeye (commercial observability). First-hire lean: start with Elementary or a few custom z-score tests; don't buy a platform on day one.

Observability & lineage

Observability = knowing the health of your data over time across five pillars: freshness, volume, schema, distribution, lineage. Lineage is the map of what depends on what.

• Why lineage matters for a solo hire: when something breaks, lineage tells you the blast radius — which dashboards and metrics are affected — so you can communicate impact accurately. dbt's DAG gives you this for free.
• Tools: dbt docs/exposures (free, comes with your project), OpenLineage, DataHub. Don't over-invest early; dbt's built-in graph covers a one-person stack.

SLAs & severity

Not all data is equally critical. Set expectations explicitly so you're not paged at 3am for a non-critical model.

• Tier your data: Tier 1 = revenue/billing/exec metrics (tight freshness, hard alerts); Tier 2 = team dashboards (looser); Tier 3 = exploratory (best-effort).
• An SLA is a promise: "the daily revenue model is fresh by 8am on business days." Publish it so stakeholders know what to expect and you know what to defend.
• Severity routing: Tier-1 failures page you; Tier-3 warnings go to a channel you check. Keeps alerting trustworthy — alert fatigue is itself a quality failure.

Running a data incident

Incidents are inevitable; how you run them defines your credibility. The sequence:

1. Acknowledge fast & communicate. "We've spotted an issue with today's revenue number; investigating, update in 30 min." Silence is what erodes trust, not the bug.
2. Contain. Mark the affected dashboard stale / hold the number so no one acts on bad data.
3. Diagnose via lineage. Walk upstream — source freshness? volume drop? schema change? a logic bug in a recent merge?
4. Fix & backfill. Idempotent pipelines (chapter 04) let you reprocess the affected partitions cleanly.
5. Add a test so it can't recur silently. Every incident becomes a new tripwire. This is how a one-person team compounds reliability.
6. Brief, blameless write-up. What broke, impact, fix, prevention. Builds organizational trust in the data function.

The "the number looks wrong" scenario (script)

A near-universal interview prompt. Don't jump to "I'd check the SQL." Show a triage framework:

1. Reproduce & quantify. "Which number, which view, how wrong, since when?" Pin it down before theorizing.
2. Is the data wrong, or the expectation? Often the number is right and the definition differs ("active" means what?). Half of these are definition mismatches — check that first; it's cheap.
3. Work the pipeline top-down: source freshness/volume → staging dedup → join fan-out → metric logic → BI-layer filter. Lineage guides the walk.
4. Reconcile to a source of truth (e.g. metered usage vs the billing system — problem 12).
5. Communicate throughout, fix idempotently, add a test.

✓ Check yourself

Answer from memory before opening each one.

1. The nightly job finished green, yet the revenue chart has been flat since 03:00. Name the two cheap checks that would have caught this, and say what each one compares.

   Show answer

   A freshness check compares the newest load timestamp with now against an agreed limit. A volume check compares today's row count with a band around the trailing average. A job can succeed while loading nothing, so success alone proves little. See freshness and volume.

2. A colleague proposes unique and not_null tests on every column of every model. What is wrong with that plan, and where do you put tests instead?

   Show answer

   It creates noise, slows builds and trains people to ignore failures. Test at the boundaries: schema and freshness at ingest, key uniqueness and enums in staging, and in marts the grain, foreign keys, business invariants and reconciliation. See where to test.

3. You need a rule that paid invoice amounts are never negative. Would you write a generic test or a singular one, and how does it signal failure?

   Show answer

   A singular test: a SQL query that selects the violating rows, so any returned row fails the build. Generic tests cover reusable properties such as unique and not_null, while business rules usually need a custom query. Set severity to error if it should block. See test types.

4. The CEO says active customers fell 12% since last week. What do you check before you open the pipeline?

   Show answer

   Reproduce and quantify it: which number, which view, since when. Then ask whether the data is wrong or the definition differs, such as a changed meaning of active or a different time zone cut-off. That check is cheap and settles many cases. See the "number looks wrong" scenario.

5. Your engineering team of six will not sign a formal data contract. What do you set up so a renamed column does not silently break revenue?

   Show answer

   A schema test at ingest that fails and notifies you as soon as a column is renamed, dropped or retyped, plus a light agreement to be told before schema changes. Move to enforced contracts as the team grows. See data contracts.

6. Revenue models and an exploratory notebook feed alerts into one channel, and people have muted it. How do you fix the alerting?

   Show answer

   Tier the data. Tier 1 (revenue, billing) pages someone with tight freshness limits, while Tier 3 warnings go to a channel checked daily. An alert that needs no action should not interrupt anyone, because alert fatigue is itself a quality failure. See SLAs and severity.

Takeaway

• Trust is what a first hire ships, so build the tripwires before a stakeholder finds the break.
• Freshness, volume and key uniqueness on the few trusted metrics catch most silent failures for little effort.
• Test at the boundaries, not on every column: ingest, staging keys, mart grain and reconciliation.
• Singular tests encode business rules as queries that return zero rows; severity decides whether a failure blocks the build.
• At a startup a contract can be a schema test plus an agreement to be told before a rename.
• Tier the data, page only for Tier 1, and treat alert fatigue as a quality failure.
• In an incident, acknowledge, contain, trace through lineage, fix, add a test and write a blameless note; for a wrong number, check the definition first.

Basic tests catch breaks; detecting drift, gating changes in CI and setting SLOs takes more. → Data Quality Advanced

Further reading

• Data tests — dbt Labs docs, 2026. Generic and singular tests, severity and store-failures options.
• Source freshness — dbt Labs docs, 2026. Declaring freshness limits on sources and running the check.
• Model contracts — dbt Labs docs, 2026. Enforcing declared column names and types on a model's output.
• Open Data Contract Standard — Bitol, Linux Foundation, 2026. A vendor-neutral format for schema, quality and SLA terms.
• OpenLineage documentation — OpenLineage project, 2026. An open standard for lineage events across tools.
• Postmortem Culture: Learning from Failure — Google SRE book, 2016. Blameless post-incident reviews and what they should contain.