Drill: Data Quality & Drift Monitoring

The prompt

The prompt has a trap built in: it's tempting to answer with a list of tools (Great Expectations, Evidently, Monte Carlo) or a list of statistical tests. The interviewer is grading whether you understand that quality is layered — schema, volume, freshness, distribution, and skew are different failures with different detectors — and that a monitor without a decision attached is a dashboard nobody reads. The reference treatment is in 10 · Data Quality & Drift.

Clarifying questions

• What changed upstream? "Do we know yet whether this was a data change, a world change, or a model change?" Frames the three root-cause families before designing detectors.
• Label delay. "How long until we know a transaction was fraudulent — chargebacks at 30–90 days?" Determines whether you can monitor precision directly or must use proxies.
• Feature inventory. "How many features, how many sources, how many are computed in real time vs batch?" Sets the scale of the monitoring surface and where skew is likely.
• Blast radius and owners. "Who owns the upstream tables — is there a data contract, and who gets paged?" Determines whether alerts can route to the right team.
• Retraining cadence and gating. "Weekly retrain — is it automatic? Is there a validation gate before promotion?" Reveals whether a bad dataset can reach production unopposed.

Pinned assumptions: ~120 features from 9 source tables and 3 streams; 40 real-time features; labels arrive 30–90 days late; weekly automatic retrain with no gate; 3 million scored transactions/day.

The five moves, applied

1 · Requirements & signal

The requirement is time-to-detect and time-to-attribute: catch a material degradation within a day, not six weeks, and tell the on-call which layer broke. Because the true label arrives late, the primary signal must be a proxy — score distribution, feature distributions, and the small set of fast labels you do have (manual reviews, confirmed fraud from the ops team) — with the slow label used to calibrate the proxies. State the SLO out loud: "material drift in any top-20 feature detected within 24 hours; precision estimated weekly from fast labels with a confidence band; a model promotion blocked if validation drops more than 2 points."

2 · Sources & labels

Enumerate the failure families, because each needs its own detector:

• Schema and semantic changes upstream — a column renamed, a unit changed from cents to dollars, an enum value added. Detected by contracts and schema checks at ingestion.
• Volume and freshness — a source stops, a partition arrives late, a stream backfills twice. Detected by row-count and freshness monitors per table and partition.
• Distribution drift — the world changed (a new merchant category, a holiday), or a pipeline bug changed the values. Detected by comparing feature distributions to a reference window.
• Training/serving skew — the offline feature and the online feature disagree for the same entity at the same time. Detected only by logging what the model actually saw at scoring time and joining it back. This is the one most teams don't have and the one that most often explains "it worked in validation".
• Label quality — the fraud team changed its review policy, so the label definition shifted. Detected by label-rate monitoring and periodic audits, and it's the family that no statistic catches without a human in the loop; see 03 · Labels & Feedback Loops.

3 · The offline path

Put checks at the hops, not only at the end. Ingestion: schema validation against the contract, null-rate and range checks, volume vs a seasonal baseline. Feature computation: per-feature statistics written to a feature stats table every run (count, null rate, mean/std or quantiles, cardinality for categoricals) — cheap, and the substrate for everything else. Training: a dataset validation gate that compares the new training set to the last accepted one (PSI or KS per feature, label rate, leakage checks such as a feature that predicts the label too well) and refuses to train on a dataset that fails. Promotion: evaluation on a frozen holdout and on the most recent labelled slice, with a hard floor.

4 · The online path

Two logs make the online side observable. First, log the feature vector as scored — every request's inputs, model version, score, and feature timestamps, to a stream that lands in the lakehouse. Second, compute online feature statistics in windows (hourly) and compare them to the training reference and to the offline recomputation of the same features for a sample of the same entities — that comparison is the skew detector. Score distribution monitoring (share of transactions above threshold, mean score by segment) is the earliest proxy for "something changed", often days before any label arrives.

5 · Operate

Every monitor has three things attached: an owner, a threshold with a rationale, and a runbook step. Alert routing follows the layer — a schema violation pages the source team via the contract; a skew alert pages ML platform; a score-distribution shift opens a triage ticket for the model owner with the top drifted features attached. Retraining is a response, not a reflex: drift caused by a pipeline bug must be fixed, not trained around. Close the loop by back-testing every detector against the six-week incident: which one would have fired first, and on what day.

The whiteboard

          ┌─ contracts + schema checks ─┐  ┌─ volume / freshness ─┐   ┌─ dataset validation gate ─┐
 sources ─┴──▶ ingestion ──▶ features ───┴──▶ training set ─────────┴──▶ train ──▶ eval gate ──▶ registry
   (9 tbl,        │            │  feature stats table (per run:            (PSI/KS vs last        (frozen holdout
    3 streams)    │            │  count, nulls, quantiles, cardinality)     accepted set,          + recent slice,
                  │            │                                            leakage checks)        hard floor)
                  │            ▼
                  │      reference window (training distribution)
                  │            │ compare (PSI per feature, hourly)
                  │            ▼
 online:  request ──▶ online features ──▶ model v ──▶ score ──▶ decision
                         │                              │
                         └── log features-as-scored ────┴──▶ scored log (lakehouse)
                                   │                            │
                                   │  sample entities             │ score distribution by segment
                                   ▼                              ▼
                        skew check: online vs offline    proxy alarms ──▶ triage ticket + top drifted features
                        recompute for same entity/time
 labels (30–90 d late) ──▶ precision/recall by cohort ──▶ calibrate proxies · audit label policy
 every alert: owner · threshold + why · runbook step · route by layer

Trade-offs table

What the interviewer is listening for

• Layers, with a detector per layer. Contracts, volume/freshness, distribution, skew, labels — not "I'd monitor drift".
• Logging features as scored. The single design element that makes skew detectable and incidents reproducible.
• Late labels handled honestly. Proxies calibrated against the slow truth, with confidence bands, not a fake real-time precision metric.
• A gate before promotion. Weekly automatic retrain with no gate is the root cause in the prompt; say so.
• Alerts that route and resolve. Owner, threshold rationale, runbook, routing by layer, and a back-test against the actual incident.

Follow-up probes

• "Labels take 60 days. How do you know precision dropped before then?" — You don't, directly. You watch score distribution by segment, alert-rate vs review capacity, fast partial labels from manual review, and feature drift on the top-importance features. When the slow labels arrive, you measure how well those proxies predicted the drop and tune them.
• "Static PSI thresholds page you 40 times a night. Now what?" — Prioritise by feature importance × drift magnitude, alert on the top-N composite, move to adaptive thresholds against a seasonal baseline, and separate "informational" from "actionable" by attaching a decision to each alert. A monitor without a decision gets deleted.
• "The drift is real — a new payment method launched. Retrain?" — Yes, but through the gate: confirm the new segment has labels or a proxy, retrain with it represented, evaluate on a slice that includes it, and monitor that slice separately after promotion. Also add the launch calendar as a known-change annotation so the alert can be explained, not just silenced.
• "Online feature shows 3% nulls, offline shows 0.1%. Which is right?" — Neither is "right"; the skew is the bug. Usually an online timeout defaulting to null. Fix the online path or teach training about the null rate it will actually see; either way the two must agree, and the skew monitor stays.
• "Where does a data contract help and where doesn't it?" — It catches schema and semantic changes at the producer boundary and routes the alert to the right owner; it does nothing for distribution shift or label drift, which are downstream and statistical.
• "Cost of logging every scored feature vector?" — 3M/day × ~120 features × ~8 bytes ≈ 3 GB/day uncompressed, under 1 GB in Parquet; trivial. Sample only if request volume is two orders of magnitude higher, and never sample the entities you use for skew checks.

The answer at four levels

The drills are done. Next: the decisions that recur across every one of them, as matrices you can defend — 08 · The Big Trade-offs.

✓ Check yourself

Answer from memory before opening each one.

1. A source team changes order amounts from cents to dollars without notice. Which detectors catch it, and which stay silent?

   Show answer

   A contract that encodes units catches it at ingestion. Without one, distribution drift on the amount features fires downstream as a collapse by a factor of 100, and the score distribution shifts. Volume and freshness monitors stay green, since rows still arrive on time. See the five moves.

2. Why can an offline recomputation of the features not detect training and serving skew by itself?

   Show answer

   It runs the offline code, so it agrees with the offline values by construction. Skew is a disagreement with what the model actually saw online, which you can only compare if you logged the vector at scoring time and join it back. See the five moves.

3. A churn model's labels arrive 90 days late. A feature pipeline was changed yesterday. What can you monitor this week, and what do the slow labels do later?

   Show answer

   Watch the feature distributions against the training reference, the score distribution by segment, and any fast partial labels, such as manual reviews. When the 90-day labels mature, measure how well those proxies predicted the change and tune their thresholds. See the five moves.

4. A drift alert fires because a pipeline bug doubled a feature, and the weekly automatic retrain runs tonight. What is wrong with letting it, and what should stop it?

   Show answer

   Retraining on the bad values launders the bug into the model and hides the cause. The dataset validation gate, which compares the new set to the last accepted one per feature and checks the label rate, should refuse to train. The response here is to fix the pipeline. See the whiteboard.

5. A retailer has strong weekly and holiday seasonality. Do you compare features to the frozen training distribution or to a rolling window?

   Show answer

   Use both for different questions. The frozen reference answers whether the world has moved since training, but seasonality will keep it alarming. A rolling or seasonal baseline answers whether something broke today. Pick by the decision the alert drives. See the trade-offs table.

6. There are 300 features with one static PSI threshold, and the on-call is paged 40 times a night. What is the structural fix, beyond raising the threshold?

   Show answer

   Rank alerts by feature importance times drift magnitude and page on the top few, use adaptive thresholds against a seasonal baseline, and attach a decision to each alert. A monitor with no decision attached gets ignored or deleted. See the follow-up probes.

Takeaway

• Quality is layered. Schema and semantic change, volume and freshness, distribution, skew and labels are different failures and each needs its own detector.
• Labels arrive late, so monitor proxies such as score distribution, feature distributions and fast partial labels, and calibrate them against the slow labels later.
• Logging features as scored is what makes skew detectable and incidents reproducible.
• Put a validation gate before training and before promotion. An automatic weekly retrain with no gate was the root cause in the prompt.
• Drift is a symptom. By layer, the response is fix, retrain or investigate, and retraining on a pipeline bug launders it into the model.
• Every monitor carries an owner, a threshold with a rationale and a runbook step, and you back-test the detectors against the real incident.

The five drills share a small set of recurring forks, which the next chapter lays out as matrices with flip conditions. → The Big Trade-offs

Further reading

• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — Breck et al., Google, 2017. A checklist of data, model and monitoring tests to score a pipeline against.
• Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift — Rabanser et al., 2018. Which drift tests work and where they fail.
• TensorFlow Data Validation — TFX documentation. Schema inference, anomaly detection and skew and drift checks between datasets.
• Great Expectations documentation — Great Expectations. Declarative data-quality checks, for the ingestion and dataset-gate layers.
• Evidently documentation — Evidently AI. Data and prediction drift reports and monitoring for deployed models.