Start Here

What this guide is (and isn't)

This is a guide about the data architecture that machine learning runs on — the pipelines that assemble training sets without leaking the future, the stores that serve features in single-digit milliseconds, the versioning that lets you reproduce a model six months later, and the monitoring that tells you the world has moved before the model's numbers do. It is written for the engineer who owns that layer: data platform engineers, ML platform engineers, and the senior data engineer who has just been told "the fraud model is wrong and nobody knows why."

It is not a guide to modeling. You will not find loss functions, architectures, or hyperparameter advice here. Models are treated as consumers of data with specific, demanding contracts — and the thesis of the whole guide is that most production ML failures are data-architecture failures wearing a model's clothes: a feature computed one way offline and another way online; a label joined a day too early; a training set that silently drifted from what the API sees; a dataset nobody can rebuild.

It is also not interview prep. That is the sibling guide, ML Data Architecture — Interview Prep, which teaches the same material as a design method with drills and graded answers. Read this one to understand the systems; read that one the week before a loop.

The running case: GridDP's four models

Throughout the guide we build the ML data plane for GridDP, the internal data platform of a hypothetical GPU-rental marketplace. It is the same case study as the sibling Data Platform Systems Design guide, and this guide assumes that platform exists: the nine source systems, the telemetry firehose (~17,000 GPUs reporting every 10 seconds, ~150 M samples a day), the canonical model with dim_account, dim_host, fct_rental, fct_bid_event, fct_telemetry_hourly, and the three reference stacks — Stack A (open-source lakehouse: Iceberg, Flink, Feast, MLflow), Stack B (Databricks-centric), Stack C (Snowflake-centric). If those names are unfamiliar, read that guide's Start Here first; it takes ten minutes and everything here builds on it.

GridDP ships four models, and they are chosen because together they exercise every part of the data plane:

Notice the spread: one batch model, one streaming model, two request-path models; labels that arrive instantly (a click) and labels that arrive in six weeks (a chargeback); features from a slowly-changing dimension and features from a firehose. When a chapter says "for the fraud model…", it is picking the case that makes the design tension sharpest.

The ML data plane: two paths and a loop

Everything in this guide fits one picture. Hold it in your head; every chapter is a zoom into one box.

                     ┌────────────────────────── THE LOOP ──────────────────────────┐
                     │                                                               │
  SOURCES            ▼            OFFLINE PATH (hours–days, throughput)              │
  ┌──────────┐   ┌────────┐   ┌──────────┐   ┌───────────┐   ┌──────────┐   ┌────────┐
  │ OLTP/CDC │──▶│ lake-  │──▶│ feature  │──▶│ training  │──▶│ training │──▶│ model  │
  │ events   │   │ house  │   │ pipelines│   │ datasets  │   │  (GPUs)  │   │registry│
  │ telemetry│   │(bronze │   │ (batch)  │   │ point-in- │   └──────────┘   └───┬────┘
  │ 3rd party│   │ silver │   └────┬─────┘   │ time, ver-│                      │ deploy
  └──────────┘   │ gold)  │        │         │ sioned    │                      ▼
       │         └────────┘        │ materialize └───────────┘   ONLINE PATH (ms, latency)
       │ streams                   ▼                              ┌──────────┐   ┌─────────┐
       └──────────────────▶ ┌────────────┐  ◀── streaming ───────│ inference │◀──│ request │
                            │ online     │      features          │ service   │   │ (search,│
                            │ store (KV) │──── feature vector ───▶│ + model   │   │ signup) │
                            └────────────┘                        └────┬──────┘   └─────────┘
                                                                       │ predictions, outcomes
              labels ◀──── ground truth arrives later (clicks, chargebacks, actuals) ◀──┘
              monitoring ◀─ drift, skew, quality — compares offline vs online, then vs now

• The offline path is optimised for throughput and correctness over time: it turns the lakehouse into versioned training datasets via feature pipelines and point-in-time joins, and feeds GPUs. Chapters 02–05 and 12.
• The online path is optimised for latency and freshness: it assembles a feature vector for one entity in milliseconds from an online store, sometimes updated by streams seconds ago, and returns a prediction. Chapters 04, 06, 07, 08.
• The loop is what makes ML data different from analytics data. Predictions change behaviour, behaviour becomes tomorrow's training data, and ground truth for today's predictions arrives later — sometimes much later. Chapters 03, 09, 10.

The single most important property of this picture is the seam between the paths: the same feature must mean the same thing on both. A feature store (ch. 04) is, at bottom, a machine for enforcing that. Training/serving skew — the value computed offline for training disagreeing with the value served online — is the defining bug of the field, and you will meet it in nearly every chapter.

Six principles the guide keeps returning to

1. Time is the primary key. Every feature value has a validity interval; every training row has a prediction timestamp; a join is only correct if it respects both. "As of when?" is the question behind most bugs.
2. Datasets are artifacts, not queries. A training set is built, versioned, hashed and registered like a binary — never "the result of running this notebook again."
3. Define once, serve twice. One feature definition feeds the offline store and the online store. Two implementations is a skew factory.
4. Labels are data with latency. Ground truth arrives late, partially, and with bias (you only see chargebacks for payments you approved). Model the delay explicitly.
5. The firehose sets the budget. At GridDP ~80% of bytes are telemetry; the ML plane inherits that 80/20. Rollups, tiering and sampling decisions are cost decisions first.
6. Reproducibility is a chain, not a tool. Code version + data version + feature definitions + environment. Break one link and you cannot explain a model's behaviour.

How the chapters fit, and reading order

Full read: 00 → 14 in order; the case compounds. "We're standing up a feature store": 01 → 02 → 04 → 08 → 10 → 13. "Our model is wrong and we don't know why": 01 → 02 (leakage) → 09 (can you reproduce it?) → 10 (drift or skew?). "Real-time fraud": 03 → 04 → 06 → 08 → 11.

Start with the lifecycle and the places it breaks: 01 · The ML Data Lifecycle & Where It Breaks.