Data Platform · Deep Dive

ML Data Architecture

The data architecture underneath production machine learning, end to end — training data and labels, features and the feature store, storage and dataset versioning, streaming, embeddings, serving and skew, then the operating discipline of lineage, quality, governance and cost. Built on one running case study and closed with three reference architectures compared.

How to read this

Start with 00 · Start Here — it sets up the case study and the one idea everything else hangs off: there are two paths (offline, for training; online, for serving) and one loop (today's decisions become tomorrow's labels). Section B builds the offline path, Section C the online path, Section D keeps both honest, and Section E assembles the whole plane. Preparing for an interview? Pair this with ML Data Architecture — Interview Prep.

Section A · Orient

00 Start Here The running case study — GridDP's four production models, the two paths (offline for training, online for serving) and the one loop that connects them, plus reading order. Read first. 01 The ML Data Lifecycle & Where It Breaks The eight stages data passes through on its way to and from a model, the two speeds they run at, who owns each, and the specific place in every stage where production ML quietly goes wrong.

Section B · The data

02 Training-Data Pipelines & Point-in-Time Correctness The entity-timestamp spine, the as-of join that keeps the future out, the four kinds of leakage, honest splits and dedup — and why a dataset is a versioned artifact, not a query. 03 Labels, Ground Truth & Feedback Loops Where labels come from, how late they arrive, what they systematically miss, and how to design the label pipeline so the model learns from the world rather than from its own decisions. 04 Feature Engineering at Scale & the Feature Store Registry, offline store and online store bound by one definition — entities and keys, point-in-time retrieval, backfills, on-demand features, build vs buy, and when you don't need one. Backbone 05 Storage for ML: Lakehouse, Formats & Dataset Versioning Which file and table formats earn their keep for ML, how to lay out object storage so GPUs don't starve, and the versioning that makes a model reproducible a year later.

Section C · Moving & serving

06 Streaming Features & Real-Time Inference Data The minority of features that must be seconds fresh — what they cost, how windows and watermarks keep them correct, and the discipline that stops them disagreeing with batch. 07 Embeddings, Vector Data & RAG Pipelines Embeddings are features whose definition is a model. Why re-embedding is a migration, where vectors should live, and why "just add a vector database" is usually the wrong first move. 08 Serving Architectures & Training/Serving Skew Batch, online and streaming inference are three data paths with three failure modes. Latency budgets, online-store choice, input assembly — then hunting the skew that breaks the model. Load-bearing

Section D · Operating it

09 Lineage, Metadata & Reproducibility Model run → feature versions → dataset version → table snapshots → sources. The chain that lets you retrain, debug, audit and roll back a model a year after it shipped. 10 Data Quality, Validation & Drift Monitoring A model never throws an exception when its data goes bad — it just gets worse. Expectations at every boundary, skew checks, drift detection with a decision attached, and data SLOs. 11 Governance, Privacy & Compliance for ML Data A model is a derived dataset compressed into weights, and every obligation follows it there — consent, purpose limitation, the deletion request, access tiers, and audit for models. 12 Cost, Scale & Feeding the GPUs The expensive resource is a GPU waiting for data. Storage tiers, the data-loading path from object store to accelerator, feature-computation cost, egress, and cost governance.

Section E · Putting it together

13 Reference Architectures Compared Warehouse-centric, lakehouse-centric and streaming-first, side by side at every layer — with a decision framework keyed to team size, latency needs and budget. Decision 14 Reference Design for GridDP, Anti-patterns & Outlook The whole ML data plane on one canvas, one fraud decision traced end to end, the stack defended layer by layer, the team and roadmap — then the anti-patterns and where this is heading.

Reading paths

Full read (new to ML data)

Read 00 → 14 in order. The case study compounds: the models named in 00 become the training sets in 02, the features in 04, the online stores in 08, and the drift monitors in 10. Don't skip 00 — every later chapter assumes it.

Senior / architect (you know the modern data stack)

Skim 00–01, then go deep on 02 (point-in-time correctness), 04 (the feature store), 05 (versioning), 08 (skew), 10 (drift) and 13 (reference architectures), finishing with 14. Those carry the decisions.

"We're standing up ML on our platform" path

00 → 04 → 08 → 10 → 13. Settle the feature-store question in 04 and the serving path in 08, know what you'll monitor from 10, then pick a stack in 13 before building anything.

Interview prep

Pair each chapter with its drill in the companion guide: 04 ↔ Design a Feature Store, 02 ↔ Point-in-Time Training Data, 06 ↔ Real-Time Features, 07 ↔ Embedding Infrastructure, 10 ↔ Quality & Drift.