Data Quality, Validation & Drift Monitoring

Why ML data fails differently

Analytics pipelines fail loudly. A broken join produces a dashboard nobody believes, someone files a ticket, the number is fixed by Friday. ML pipelines fail silently and continuously: a feature pipeline that starts emitting nulls for 4% of accounts doesn't crash training — it produces a model that is 0.7 AUC points worse, and that model ships, and it ranks offers slightly wrong for three weeks until a product manager asks why conversion dipped. The system stays green the entire time.

Three properties of ML data make this worse than the analytics case. First, the consumer is a function, not a person — there is no human in the loop to raise an eyebrow at an implausible value. Second, the failure surface is statistical: nothing is wrong with any single row, the distribution just moved. Third, there are two pipelines that must agree — the offline path that trained the model and the online path that feeds it — and they are usually written by different people in different languages.

So quality for ML has four layers, and mature platforms run all four: validation (is this batch well-formed?), monitoring (are the pipes flowing?), skew detection (do offline and online agree?), and drift detection (has the world moved since we trained?). Each catches a class of failure the others cannot.

Validation at the boundaries

Validation is the deterministic layer: assertions you can state up front and check on every batch. The mistake is to put them in one place. Put them at every boundary where ownership changes hands — source → bronze, silver → feature table, feature table → training set, training set → model. A check that lives only at training time catches the problem months after it entered the lake.

The three tools most teams end up with divide cleanly. dbt tests are the right home for anything expressible as SQL over a warehouse table — uniqueness, not-null, accepted values, relationships, and custom singular tests. Great Expectations (or Soda) is for richer, versioned suites with distributional expectations (quantiles, value sets, regex on strings) and human-readable data docs. Pandera is for in-process DataFrame validation inside Python pipelines — the dataset builder, the batch scorer, the feature-transform step — where a round trip to the warehouse is the wrong shape.

import pandera as pa
from pandera import Column, Check

# The contract for the fraud training set, checked every time the builder runs.
fraud_training_schema = pa.DataFrameSchema(
    {
        "account_id":            Column(str, nullable=False),
        "prediction_ts":         Column("datetime64[ns, UTC]", nullable=False),
        "label_fraud_30d":       Column(int, Check.isin([0, 1])),
        "payment_method_age_d":  Column(float, Check.ge(0), nullable=True),
        "device_velocity_24h":   Column(int, Check.between(0, 500)),
        "kyc_outcome":           Column(str, Check.isin(["pass", "review", "fail", "unknown"])),
        "chargebacks_90d":       Column(int, Check.ge(0)),
    },
    checks=[
        # No feature may be observed after the prediction timestamp (point-in-time correctness).
        Check(lambda df: (df["feature_max_event_ts"] <= df["prediction_ts"]).all(),
              error="feature values observed after prediction_ts — leakage"),
        # Class balance must stay inside the envelope the model was designed for.
        Check(lambda df: 0.002 <= df["label_fraud_30d"].mean() <= 0.03,
              error="fraud base rate outside [0.2%, 3%] — check label pipeline"),
    ],
    unique=["account_id", "prediction_ts"],
    strict=True,   # any unexpected column is a failure, not a warning
)

df = fraud_training_schema.validate(build_training_frame(as_of="2026-09-01"))

Schema, volume & freshness monitors

Validation asks "is this batch right?"; monitoring asks "did the batch arrive at all, on time, at the expected size?" These are cheap, table-level signals and they catch the majority of production incidents — because most incidents are not subtle statistical drift, they are a pipeline that stopped, doubled, or changed shape.

• Freshness. Max event_time (not load time) per feature table versus now, against a declared SLA. A feature with a 1-hour SLA that is 6 hours old is serving stale values to every online request, and the model happily consumes them.
• Volume. Row count per partition against a seasonal envelope — a 7-day rolling median ± tolerance, not a fixed number. GridDP's telemetry rollup drops 40% every weekend as interruptible instances get preempted; a naive threshold pages every Saturday.
• Schema. Column set, dtypes, and nullability compared to the registered contract. Upstream services rename fields constantly; the failure mode is a feature that silently becomes all-null because kyc_result became kyc_outcome.
• Null and default rates. Per column, the share of nulls and of sentinel defaults (0, -1, "unknown"). A jump in defaults is an upstream outage wearing a disguise.

Tooling here is a spectrum: dbt source freshness and simple metric queries scheduled by the orchestrator at the cheap end; Monte Carlo, Elementary, Bigeye, or Datadog data monitors when you want anomaly detection and lineage-aware alert routing without writing it yourself. The platform-engineering answer is to emit every monitor as a metric (freshness seconds, row count, null rate) into the same observability stack that watches the services, so an ML data incident and a service incident are visible on one pane.

Training–serving skew

Skew is the failure that is unique to ML: the model was trained on features computed one way and is served features computed another way. The same feature name, two implementations, drifting apart. Classic sources:

• Two codebases. A dbt model computed acct_rentals_30d for training; a Go service computes it for serving with a different definition of "30 days" (calendar vs rolling, inclusive vs exclusive of today).
• Time semantics. Offline features are point-in-time joined at prediction_ts; online features are read now — and "now" includes events that arrived in the last few minutes that the training data never saw at that horizon.
• Missing-value handling. Training imputed nulls with the column median; serving substitutes 0.
• Encoding. A categorical feature's vocabulary is frozen at training time; a new GPU model (B300) appears in production and maps to "unknown" — or worse, to an index that collides.

The structural fix is the one chapter 04 built: one feature definition, materialized to both stores from the same transformation. But definition-level unity is not enough; you still need a skew check in production. The pattern is to log the exact feature vector the model saw at inference time (the "served features" log), then periodically recompute the same features offline for the same entities and timestamps and compare — per feature, a match rate for categoricals and a distribution distance for numerics. That log is also what makes the next chapter's audit story possible.

-- Compare what the ranking model was SERVED with what the offline pipeline says it SHOULD have seen.
WITH served AS (
  SELECT offer_id, served_at, feature_json:host_reliability_7d::FLOAT AS served_rel
  FROM ml.served_features_log
  WHERE model = 'ranking_v14' AND served_at >= DATEADD('day', -1, CURRENT_TIMESTAMP)
),
recomputed AS (
  SELECT s.offer_id, s.served_at, f.value AS offline_rel
  FROM served s
  ASOF JOIN feature_store.host_reliability_7d f
    MATCH_CONDITION (s.served_at >= f.valid_from) ON s.offer_id = f.offer_id
)
SELECT
  COUNT(*)                                                     AS n,
  AVG(ABS(served_rel - offline_rel))                           AS mean_abs_diff,
  AVG(CASE WHEN ABS(served_rel - offline_rel) > 0.05 THEN 1 ELSE 0 END) AS share_mismatch
FROM served JOIN recomputed USING (offer_id, served_at);
-- Alert if share_mismatch > 2%: something in the two paths has diverged.

Drift: feature, prediction, concept

Drift is the world changing after training. Three kinds, and they need different detectors and different responses:

Population Stability Index is the workhorse for tabular features because it is bucketed, cheap, and has conventional thresholds (below 0.1: stable; 0.1–0.25: investigate; above 0.25: significant shift). For each feature, bucket the training baseline into deciles, count the live population into the same buckets, and compute Σ (live% − base%) · ln(live% / base%). The Kolmogorov–Smirnov test is the alternative for continuous features when you want a statistic that doesn't depend on binning. For embeddings — the ranking model's offer embedding, the support-ticket text embedding — bucketing is meaningless; monitor the distribution of distances to a reference centroid, or train a small domain classifier (baseline vs live) and alarm when it can separate them.

Delayed labels are the hard part of concept drift. GridDP's fraud label — a chargeback or confirmed abuse — arrives 30–90 days after the prediction. So the fraud model's real accuracy is always a quarter old, and feature/prediction drift are the only leading indicators you have. Design for that: the monitoring dashboard shows leading signals (drift) beside the lagging one (AUC on matured labels), and the retrain cadence is decided on both.

Data SLOs & incident response

None of the above matters unless it pages the right person with enough context to act. Treat ML data like a service with SLOs: declare them per feature table and per model, measure them continuously, and run incidents when they're breached. A workable set for a tier-1 model:

• Freshness SLO: 99% of online feature reads served with feature age ≤ TTL (e.g. 15 min for bid_depth_5m, 24 h for host_reliability_7d).
• Completeness SLO: ≤ 0.5% of inference requests fall back to default values for any single feature.
• Skew SLO: served-vs-offline mismatch ≤ 2% per feature, checked daily.
• Drift budget: no tier-1 feature above PSI 0.25 for more than 24 h without an open investigation.
• Baseline SLO: every deployed model has a recorded training-distribution baseline; a model without one cannot be promoted.

Incident response for data incidents has one difference from service incidents: the blast radius is in the past. When a feature pipeline is found to have been wrong for nine days, the fix is not just to repair the pipeline — it is to decide whether the model trained during those nine days is contaminated, whether the served-feature log needs annotating so future skew analysis excludes the window, and whether downstream metrics (conversion, fraud loss) need restating. The runbook should force those three questions explicitly.

Worked example: the fraud model goes quiet

Week one: GridDP's KYC vendor ships an API change. The field result, previously pass | review | fail, now returns PASS | REVIEW | FAIL | PENDING. The ingestion job lands the new strings. The feature pipeline's mapping to kyc_outcome matches none of them and falls through to "unknown". Nothing errors.

What each layer would have caught:

1. Boundary validation at source → bronze — an accepted-values test on result — fails on the first batch. Incident opened in minutes, routed to the integration owner. This is where it should stop.
2. Without that, the null/default-rate monitor on kyc_outcome sees "unknown" jump from 1% to 100% within an hour.
3. Without that, feature drift on kyc_outcome shows PSI > 1.0 at the next daily run.
4. Without that, prediction drift: the fraud score distribution compresses toward the middle because a strong signal became uninformative — visible the same day.
5. Without any of it, concept drift shows up 60 days later as a fraud-loss spike in finance's reconciliation, and the model has meanwhile been retrained on a quarter of "unknown" KYC.

The costs escalate by roughly an order of magnitude per layer skipped. That is the argument for spending on the boring, deterministic checks first.

When to choose what

Next: the data you are allowed to train on at all, and what you owe the people it describes — 11 · Governance, Privacy & Compliance for ML Data.

✓ Check yourself

Answer from memory before opening each one.

1. A feature pipeline starts emitting nulls for 4% of accounts. Training does not crash, and the model ships. Why does ML data fail this way when an analytics table does not, and what do you build for it?

   Show answer

   The consumer is a function, not a person, so no one notices an implausible value, and the failure is statistical, since no single row is wrong. Treat each dataset as an API with a contract on schema, row count, freshness and distributions, and check it at every ownership boundary. See why ML data fails differently.

2. Where should the first check on the KYC vendor's response sit, and why does a check only at training time arrive too late?

   Show answer

   At source to bronze, as an accepted-values test on the field, with the page routed to the producing team. A training-time check catches the problem months after it entered the lake, after the feature table has already been built and served on bad values. See validation at the boundaries.

3. The telemetry rollup drops 40% every weekend and a fixed row-count threshold pages every Saturday. What do you use instead?

   Show answer

   A seasonal envelope, such as a rolling seven-day median plus or minus a tolerance, per partition. A fixed number cannot tell a weekend from an outage. See schema, volume and freshness monitors.

4. Your daily skew query shows 6% of served host reliability values differ from the offline recomputation by more than 0.05. State what the check measures and what you do next.

   Show answer

   It compares the logged served vector with the value an as-of join gives for the same entity and time, per feature, so 6% mismatch against a 2% tolerance means the two paths diverged. Find which definition, time semantics, missing-value or encoding rule differs and fix the shared definition. See training-serving skew.

5. Device velocity shows PSI above 0.25. A colleague wants an automatic retrain. What do you check first, and why?

   Show answer

   Check lineage for an upstream change, since drift says the distribution moved and not why. If a vendor changed its algorithm, retraining teaches the model the vendor's bug, and if a real fraud ring arrived, retraining is right. See drift: feature, prediction, concept.

6. Fraud labels arrive 30 to 90 days late. What do you put on the dashboard to know the model is healthy now, and how do you decide when to retrain?

   Show answer

   Show feature and prediction drift beside accuracy on matured labels, because drift is the leading signal and matured labels are the lagging truth. Decide retrain cadence from both. See drift: feature, prediction, concept.

Takeaway

• ML data fails silently, so treat every dataset a model consumes as an API with a contract, and treat anything that breaks it as a breaking change even if each row is valid.
• Put deterministic checks at every ownership boundary, and use strict schemas so an unexpected column has to be explained.
• Freshness, volume, schema and default-rate monitors catch most incidents, and volume thresholds need a seasonal envelope.
• Skew is checked by logging the served vector and recomputing offline for the same entities and times, with a stated tolerance.
• Feature, prediction and concept drift need different detectors, and a drift alert triggers a lineage lookup before any retrain.
• Write SLOs for freshness, completeness, skew, drift and baselines, and remember a data incident's blast radius is in the past.

Once the data is trustworthy, the next question is whether you may use it at all. → Governance, Privacy & Compliance for ML Data

Further reading

• Data Validation for Machine Learning — Breck et al., SysML 2019. The schema, skew and drift checks used in production at Google, and why they were built.
• TensorFlow Data Validation: getting started — TensorFlow documentation, accessed 2026. Statistics, schema inference, and skew and drift comparison between datasets.
• Great Expectations documentation — Great Expectations, accessed 2026. Expectation suites and validation results for batch data checks.
• Pandera documentation — Pandera, accessed 2026. DataFrame schemas with column checks and strict mode.
• Learning under Concept Drift: A Review — Lu et al., IEEE TKDE, 2019. A survey of drift types, detection and adaptation.
• Evidently documentation — Evidently AI, accessed 2026. Data drift and model quality reports, including the tests used per column type.