Section A · Orient

The ML Data Lifecycle & Where It Breaks

The eight stages data passes through on its way to and from a model, the two speeds they run at, the roles that own them, and the specific place in each stage where production ML quietly goes wrong.

Eight stages, one loop

Analytics data has a direction: sources flow into a warehouse and out to dashboards. ML data has a loop, and the loop is why it needs its own architecture. Here are the stages, in the order data meets them at GridDP, with the artifact each one produces. Every later chapter is about one or two of these boxes.

lifecycle.txt

  stage        produces                       GridDP example                         chapter
  ─────────    ───────────────────────────    ────────────────────────────────────   ───────
  1 collect    raw events, CDC, telemetry     fct_rental, bid events, GPU samples     (DPE 04)
  2 label      ground truth per entity/time   chargeback? churned? clearing price     03
  3 features   named, typed, timestamped      host_reliability_7d, bid_depth_5m       04, 06
               values per entity
  4 assemble   versioned training dataset     fraud_train_v14 (rows = payment,        02, 05
               with point-in-time joins       prediction_ts, label, features)
  5 train      model artifact + lineage       fraud-xgb:2026-09-01 in the registry    09, 12
  6 serve      feature vector + prediction    <100 ms P(fraud) on /payments           04, 07, 08
  7 observe    predictions, outcomes, drift   score distribution, PSI, skew alerts    10
  8 feed back  new labels + behaviour → 1     approved payments become next month's   03, 09
                                              training rows (with survivorship bias)

Two things to notice. First, stages 1–5 are batch-shaped — they care about throughput and correctness across history — while stage 6 is request-shaped — it cares about milliseconds and freshness. Second, stage 8 closes the loop back to stage 1, which is the difference between a data pipeline and a control system. A model that ranks listings changes which listings get clicked, which changes the clicks that train the next ranker. Ignore the loop and you will train on your own predictions.

The two-speed problem

The defining architectural tension is that the same feature must exist at two speeds. Take bid_depth_5m — the number of interruptible bids on a GPU class in the last five minutes, used by both the pricing model and the fraud model.

Dimension	Offline (training)	Online (inference)
Question	What was `bid_depth_5m` for every (GPU class, timestamp) in the last 18 months?	What is `bid_depth_5m` for H100-80GB right now?
Shape	Hundreds of millions of rows, window-aggregated over a history table	One key lookup
Latency budget	Minutes to hours	< 10 ms for the lookup, so the whole request stays < 100 ms
Freshness	Whatever the batch cadence is — the value as of each historical timestamp	Seconds — computed from the live bid stream
Computed by	SQL / Spark window over `fct_bid_event`	Flink job keyed by GPU class, sliding window
Stored in	Lakehouse (Iceberg / Delta), full history	Redis / DynamoDB, latest value with a TTL

Two engines, two stores, two codepaths — and the model was trained on the left column and is served from the right. If the SQL window counts bids with status IN ('open','matched') and the Flink job counts only 'open', the model sees a systematically different world in production than it was trained on. That is training/serving skew, and it is invisible in every offline metric. Every architecture in this guide is, in part, a strategy for collapsing the two columns into one definition. Chapters 04 and 06 are the mechanics.

The skew you can't see

Skew never fails loudly. The API returns valid floats, the model returns valid scores, and precision drifts down 4 points over a quarter. The only defence is measurement: log the served feature vector alongside the prediction, replay the offline pipeline for the same entities and timestamps, and compare. Chapter 10 builds that comparison; chapter 08 explains why you must log the vector, not recompute it.

Who owns which stage

ML data architecture fails at seams between teams as often as at seams between systems. A workable division at a company of GridDP's size (tens to low hundreds of internal data users):

Role	Owns	Their contract with the rest
Data platform engineering	Collect, the lakehouse, CDC/streaming, storage, governance, the feature store infrastructure	Freshness and quality SLOs on silver/gold tables; the registry, offline/online stores and their SLAs
ML platform engineering	Feature pipeline framework, training-data assembly, registry, serving infrastructure, monitoring	"Define a feature here and it will be served consistently"; reproducible training datasets
Data scientists / ML engineers	Feature definitions, labels, models, evaluation	Features registered with owner, entity, TTL, tests; labels documented with their delay
Product / domain teams	Source events and their schemas (data contracts)	Don't change an event's meaning without a version bump; the ranker depends on it

The failure pattern is when the middle row is missing: data scientists each build their own feature pipelines from raw tables, the platform team has no idea what depends on fct_rental, and the first schema change breaks three models. The feature registry (ch. 04) is as much an organisational device as a technical one — it is the place where a feature acquires an owner.

Where each stage breaks

Each stage has a characteristic failure. Knowing them lets you diagnose a "the model is wrong" report in minutes instead of days, because you know which stage to interrogate first.

Stage	Characteristic failure	How it presents	Defence (chapter)
Collect	Silent schema or semantic change upstream (a field's unit changes from seconds to ms)	Feature distribution shifts on a deploy date	Data contracts, schema registry, freshness/volume monitors (10; DPE 09–10)
Label	Delayed or censored labels; label leakage into features	Offline AUC 0.97, production precision collapses	Explicit label delay, censoring windows, leakage tests (02, 03)
Features	Two implementations of one feature (skew); features computed from data not yet available at prediction time	Offline/online value mismatch; step change in score distribution	Single definition, point-in-time semantics, skew monitoring (04, 06, 10)
Assemble	Non-reproducible training set; duplicate rows across splits; random split on time-series data	Can't rebuild last quarter's model; wildly optimistic validation	Datasets as versioned artifacts, time-based splits, dedup by entity (02, 05)
Train	Data loading starves GPUs; wrong dataset version picked up	10% GPU utilisation; "it was fine on my run"	Sharded, streaming loaders; registry links model → dataset hash (09, 12)
Serve	Stale or missing online features; feature assembly recomputes logic in the API	Defaults (zeros) served silently; p99 blows the latency budget	Online store SLAs, null-rate alerts, served-vector logging (08, 10)
Observe	Monitoring the model but not the data; no baseline	Drift discovered by a customer	Feature-level drift + skew + quality SLOs (10)
Feed back	Survivorship bias; training on own predictions	Model reinforces its own decisions; fraud model never sees fraud it blocked	Hold-out / exploration traffic, delayed-label design, bias-aware sampling (03)

Worked example: the fraud model's first outage

Six weeks after GridDP's fraud model ships, chargebacks on approved payments double. The model's offline metrics are unchanged. The on-call ML engineer walks the stages:

Collect. Volume and schema monitors on payments are green. The upstream payments provider did, however, start returning card_bin as an integer instead of a zero-padded string on 1 September.
Features. bin_country is derived from a lookup on card_bin. Offline, the batch pipeline casts to string and pads — the lookup still works. Online, the Flink enrichment does a raw string match on "0412…" vs 412… — the lookup misses and the feature is served as null, which the model's imputer turns into "unknown country", a low-risk bucket.
Serve. The null-rate on bin_country went from 0.3% to 41% on 1 September. There was no alert, because the null-rate monitor watched the offline table, which was fine.

Nothing about the model was wrong. The feature had two implementations (two-speed problem), the upstream contract changed (collect), and monitoring watched the wrong side of the seam (observe). Three chapters of this guide exist because of exactly this story.

The reframe

"The model is wrong" is almost never a modeling statement. It is a hypothesis to be falsified stage by stage, starting from the seams: offline vs online, then vs now, features vs labels. Build the plane so those comparisons are cheap.

When to choose what

How much of this plane do you need?

One batch model, scored nightly (capacity forecasting alone): you need stages 1–5 and 7. No online store, no streaming. A lakehouse, point-in-time-correct SQL, versioned datasets and a drift check are enough. Don't buy a feature store yet.
A request-path model with batch features (ranking with daily-refreshed features): add stage 6 with a materialised online store. Batch → KV sync is the whole "streaming" story.
A request-path model with fresh features (fraud with velocity features, pricing): now you need the streaming path, a registry that enforces one definition across engines, and skew monitoring. This is where a feature store earns its keep.
Several models sharing features across teams: the registry becomes organisational infrastructure; invest in discovery, ownership and tests (ch. 04, 09).

Optional: the lifecycle answer at four levels

L1 · Foundational. Describes ML data as "collect, clean, train, deploy"; doesn't distinguish offline from online; would recompute features in the API.
L2 · Practitioner. Names the offline/online split and training/serving skew; has hit leakage once; monitors model metrics but not feature distributions.
L3 · Senior. Walks the eight stages with the characteristic failure of each; designs the seams to be measurable (served-vector logging, replay); assigns ownership per stage.
L4 · Staff. Treats the loop as a control system — feedback, survivorship bias, exploration traffic — and sizes how much of the plane a given product needs, with the flip conditions for adding streaming and a registry.

Next, the stage where the most expensive mistakes hide: 02 · Training-Data Pipelines & Point-in-Time Correctness.

✓ Check yourself

Answer from memory before opening each one.

A model's offline AUC is unchanged, yet chargebacks on approved payments doubled six weeks after launch. You can look at only one comparison first. Which, and why?

Show answer
Compare the feature values the API served with the values the offline pipeline computes for the same entities and timestamps. A model that scores well offline and fails live usually sees a different world online, so the break is at the seam between the two paths, not inside the model. Monitoring that watches only the offline table cannot see it. See the two-speed problem.
The SQL window for bid_depth_5m counts bids with status open or matched, and the Flink job counts only open. Every API call succeeds and every score is a valid float. What fails, and what is the only reliable way to find out?

Show answer
Training and serving skew: the model trains on one definition and runs on another, so precision erodes with no error anywhere. Offline metrics cannot show it. Log the served feature vector with each prediction, replay the offline pipeline for the same keys and times, and compare. See the two-speed problem.
Why does the guide call stage 8 the difference between a data pipeline and a control system, and what goes wrong if you ignore it in a ranking model?

Show answer
Stage 8 feeds the model's own decisions back in as tomorrow's training rows. A ranker changes which listings get shown, so the clicks it later trains on were shaped by its earlier choices. Ignore the loop and the model learns its own behaviour. See the eight stages.
A card-processor change turns card_bin from a zero-padded string into an integer. The batch pipeline casts and pads, and the Flink job matches raw strings. Which stages failed, and which team's contract was broken?

Show answer
Collect failed first, because an upstream field changed type without a version bump, which breaks the product or provider team's contract. Features failed second, because two implementations handled the change differently, and the online lookup missed and served null. Observe failed third, because the null-rate monitor watched the table that stayed healthy. See where each stage breaks.
You inherit one nightly capacity forecaster and a request-path fraud model with five-minute velocity features. Your manager wants one feature store for both. How do you decide?

Show answer
The forecaster needs lakehouse tables, point-in-time SQL, versioned datasets and a drift check, and no online store. Fraud needs the streaming path, one definition enforced across engines, and skew monitoring, so a feature store earns its cost there. Build for fraud and let the forecaster join later if it shares features. See when to choose what.
Data scientists each build feature pipelines from raw tables, and a schema change in one source breaks three models. Which role is missing, and what artifact gives a feature an owner?

Show answer
ML platform engineering is missing: nobody owns the rule that a feature defined once is served consistently. The feature registry gives each feature an owner, entity, TTL and tests, and it shows what depends on a source table before it changes. See who owns which stage.

Takeaway

ML data runs as a loop of eight stages, so the model's own decisions shape the data it trains on next.
Every feature exists at two speeds, offline over history and online as a key lookup, and the model trains on one while it serves from the other.
Training and serving skew never raises an error, so the only defence is measuring served values against offline values for the same keys and times.
Each stage has a characteristic failure, which lets you diagnose a wrong model by walking the stages from the seams: offline against online, then against now, features against labels.
Ownership fails at the seams between teams as often as between systems, and the feature registry is how a feature acquires an owner.
Size the plane to the product: a nightly model needs no online store, and fresh request-path features are what justify streaming, a registry and skew monitoring.

The first place the future leaks in is the training set itself. → Training-Data Pipelines & Point-in-Time Correctness