Reference Design for GridDP, Anti-patterns & Outlook

The plane, in one breath

We began with the lifecycle and its two-speed problem: an offline path that must be correct and reproducible, an online path that must be fast and fresh, and the requirement that the two agree. We built training sets as point-in-time-correct, snapshot-pinned artifacts; treated labels as delayed, versioned data with their own pipelines; put a feature store — registry, offline history, online latest — at the centre; chose open formats with snapshots as the substrate; added a streaming lane for the few features that need seconds; gave embeddings the same discipline as tabular features; assembled serving so the model sees one feature vector regardless of path; made lineage the connective tissue; and wrapped it in quality, governance, and cost discipline. Chapter 13 settled the shape: lakehouse-centric with a streaming lane. This chapter draws it.

The reference architecture

┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│                          GRIDDP ML DATA PLANE  (lakehouse-centric + streaming lane)             │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
 SOURCES (DPE ch.01)     ANALYTICS PLATFORM (DPE)       ML DATA PLANE (this guide)          CONSUMERS
 ──────────────────      ────────────────────────       ──────────────────────────          ─────────
 marketplace-api ─┐                                     ┌────────────────────────────┐
 billing-service ─┤ CDC   ┌──────────────────┐  batch   │ FEATURE REGISTRY (ch.04)   │
 orchestrator ────┼──────▶│ LAKEHOUSE        │─────────▶│ name·entity·dtype·TTL·tag  │
 ranking-service ─┤       │ Iceberg on S3    │ dbt/Spark│ owner·consumers·cost       │
 trust-safety ────┘       │ bronze→silver→   │ features └─────┬──────────────┬───────┘
                          │ gold (SCD2)      │                │ governs      │
 mkt-events ──────┐ Kafka │                  │      ┌─────────▼────────┐ ┌───▼──────────────┐
 payments ────────┼──────▶│  ◀── history ────┼──────│ OFFLINE STORE    │ │ ONLINE STORE     │
 support ─────────┘       └──────────────────┘      │ feature history  │ │ Redis (inference │
                                ▲                   │ (Iceberg, PIT)   │ │ region) ≤ 5 ms   │
 host-agent ⚡FIREHOSE ──▶ ClickHouse rollups ───────│                  │ └───▲──────▲───────┘
   ~150M/day               (per-GPU-hour)           └───┬──────────────┘     │      │
                                                        │ PIT join          materialize   │ stream
 STREAMING LANE (ch.06)                                 ▼ + labels (ch.03)   (hourly)  │ writes
 Kafka ─▶ Flink: bid_depth_5m, device_velocity_24h ─────┼──────────────────────────────┘
          ip_velocity_1h, session_recency … ─▶ history to lake ──▶ (offline store, same defs)
                                                        │
                    ┌───────────────────────────────────▼───────────────────────────────┐
                    │ TRAINING-SET BUILDER (ch.02)  purpose/consent filter (ch.11)      │
                    │ → snapshot-pinned, sharded Parquet/MDS in s3://griddp-ml/datasets │
                    │ → dataset card · Pandera contract · drift baseline (ch.10)        │
                    └───────────────┬───────────────────────────────────────────────────┘
                                    ▼ shards streamed to GPU region (ch.12)
                    ┌───────────────────────────────┐   ┌──────────────────────────────┐
                    │ TRAINING (GPUs) + MLflow      │──▶│ MODEL REGISTRY               │
                    │ dataset snapshot ↔ model ver  │   │ model card · baseline · tier │
                    └───────────────────────────────┘   └───────────────┬──────────────┘
                                                                        ▼ deploy
 LINEAGE (ch.09) DataHub: column → feature → dataset → model → endpoint   ┌──────────────────┐
 MONITORING (ch.10): freshness · volume · skew · PSI · matured-label AUC  │ INFERENCE        │──▶ ranking
 GOVERNANCE (ch.11): tags · consent · retention · decision log · RLS      │ services         │──▶ fraud
 ORCHESTRATION: Dagster runs features, builders, retrains, expiry         │ served-feature   │──▶ pricing
                                                                          │ log ─▶ lake      │──▶ forecast
                                                                          └──────────────────┘──▶ churn

Read it as three horizontal bands. The analytics platform (left-centre) is the DPE guide's design and is the ML plane's upstream — every feature is derived from silver/gold or from the same Kafka topics. The ML data plane (centre-right) is this guide: registry, two stores, streaming lane, builder, training, model registry. The cross-cutting rail (bottom) — lineage, monitoring, governance, orchestration — wraps every box. The two structural decisions, visible as shapes: the streaming lane is a narrow lane for six features, not a second platform; and the served-feature log flows back into the lake, closing the loop that makes skew checks, audits and the next training set possible.

One fraud decision, end to end

Account acct_7a adds a new payment method and attempts a $1,800 reserved rental. Follow the data:

1. T0 — events. payment_method.added and rental.requested land on Kafka; the payments webhook delivers the vendor's tokenized card reference and risk score.
2. T0 + 200 ms — streaming features. Flink updates device_velocity_24h (3 → 4 accounts on this device fingerprint), ip_velocity_1h, and payment_method_age_d (0.0) in Redis, and appends the same values with event time to the lake.
3. T0 + 250 ms — inference. The fraud service assembles the vector: streaming features from Redis; batch features (chargebacks_90d, kyc_outcome, acct_tenure_d) from Redis, materialized hourly from the offline store; two on-demand features computed from request context. Schema-validated by the Pandera contract. Model fraud_v22 scores 0.83 → review. The full vector, score, model version and request ID go to the served-feature log.
4. T0 + 1 h — batch. The hourly Dagster run recomputes account aggregates from silver, materializes to Redis, and appends history to the offline store. Freshness and volume monitors pass; PSI on device_velocity_24h is 0.08.
5. T0 + 1 d — skew check. The daily job recomputes yesterday's served vectors from the offline store: 99.4% match; the mismatch is entirely ip_velocity_1h late-arriving events, within the 2% SLO.
6. T0 + 45 d — label. A chargeback arrives via the payments webhook → label_fraud_30d = 1 for the prediction at T0, written by the label pipeline with its own event time.
7. T0 + 60 d — training. The monthly builder assembles the fraud training set: labels joined point-in-time to feature history (only values known at T0), consent and jurisdiction filters applied, the deletion list honoured, contract validated, dataset card written, shards streamed to the GPU region. fraud_v23 trains at 84% GPU utilization, is registered with its dataset snapshot and drift baseline, passes champion/challenger on matured labels, and deploys.
8. T0 + 60 d + audit. A regulator asks why acct_7a was sent to review. The decision log returns the exact vector and model version; lineage walks each feature to its source columns; the dataset card for fraud_v22 shows the consent policy it was trained under. One query each.

Stack choices, defended

The team that runs it

The plane at GridDP scale is run by a small team with sharp boundaries. An ML platform engineer (one, later two) owns the registry, the stores, the builder, the loading path and the monitors — they are the DPE of the ML plane and sit with the data platform team. Data scientists / ML engineers (five-ish) own feature definitions, labels and models for their domain, and consume the platform through the registry and the builder; they never write to the online store directly. The data platform team owns everything upstream — silver, gold, Kafka, ClickHouse — and the contracts the ML plane depends on. Trust & safety and finance own label semantics (what counts as fraud, what counts as churn) and sign the model cards for their models. The failure mode to design against is the ML engineer who, under deadline, builds a private feature pipeline in a notebook; the platform's job is to make the registry path faster than the shortcut.

Crawl → walk → run

Capacity & cost math

Order-of-magnitude sizing for the plane described, at 17,000 GPUs, 200k accounts, five production models:

• Offline store: ~40 tier-1 features × hourly grain × (17k GPUs + 200k accounts + 60k offers) ≈ 2.4B feature-rows/year ≈ 0.9 TB/year in Iceberg; 3-year retention ≈ 3 TB warm.
• Online store: ~280k entities × ~40 features × ~80 bytes ≈ 1 GB; a small Redis with replicas. Peak reads: ranking at ~2k req/s × 20 features = 40k key reads/s — pipeline them, 5 ms budget.
• Streaming lane: ~30M events/day on the relevant topics; one Flink job, 8–12 task slots, ~50 GB state; always-on cost ≈ $1.5–2.5k/month.
• Training: monthly retrains of five models + experiments ≈ 900 GPU-hours/month at ≥ 80% utilization; datasets replicated once per version to the GPU region (~2 TB/month egress ≈ $150).
• Served-feature log: ~5M predictions/day × ~2 KB ≈ 10 GB/day ≈ 3.6 TB/year; warm for 90 days, cool thereafter, decision records for money models archived five years.
• Total ML data plane run-rate: roughly $8–12k/month excluding GPU time, dominated by the streaming lane and materialization compute; the GPU line is 3–4× that and is where the optimisation effort belongs.

Anti-patterns

• The notebook feature pipeline. A feature computed in a notebook for training and re-implemented in a service for serving. The whole guide is the antidote.
• Training on "now." Joining current feature values to historical labels. Every offline metric is a lie until the join is point-in-time.
• The warehouse dataloader. Querying the warehouse per epoch. Pay once, write shards.
• Streaming everything. An always-on Flink job for a churn feature that changes weekly.
• A model with no baseline. If you did not record the training distribution, you cannot detect drift; you are waiting for a complaint.
• The unowned label. Nobody can say precisely what "fraud" means in the label table, so the model learns the label pipeline's bugs.
• Vector DB as source of truth. Embeddings that cannot be regenerated because the text was deleted and the model version was not recorded.
• Deletion that stops at the tables. Erasure that never reaches the feature store, the served log, the vector index or the snapshots.
• Two truths. A separate "ML lake" that drifts from the analytics lake until the churn feature and the churn dashboard disagree in a board meeting.
• Unmeasured GPUs. Training jobs at 35% utilization for a year because nobody plotted dataloader wait time.

Where this is heading

Lakehouse-native ML. The gap between "the lake" and "the ML plane" is closing: table formats are growing vector and tensor types, catalogs are absorbing the feature and model registries, and the training-set builder is becoming a catalog operation rather than a separate tool. Expect the W-vs-L distinction to blur as warehouses adopt open formats and lakehouses adopt warehouse ergonomics.

Real-time by default, priced honestly. Streaming engines are getting cheaper to operate (serverless Flink, RisingWave, Materialize-style incremental views), so the "S lane" widens — but the discipline of asking which features actually pay for seconds will matter more, not less, as the default gets easier.

LLM and RAG data. The same architecture applies to retrieval corpora: chunk pipelines are feature pipelines, embeddings are features with a model version, retrieval logs are served-feature logs, and eval sets are training sets with point-in-time discipline. The teams treating RAG data as a side project are rebuilding this guide badly; the ones treating it as ML data are reusing it.

Agents as consumers and producers. Agentic systems will both consume the plane (feature and metric lookups as tools, governed by the same registry ACLs) and produce data for it (action logs, tool-call traces as training signal). The decision log becomes the central artifact — for audit, for training, and for the next generation of models learning from the last generation's decisions.

The constant through all of it is the thesis of this guide: a model is only as good as the data plane's ability to say, for any prediction, exactly what the model saw, why it was allowed to see it, and how to see it again.

What to read next

This guide is the reference. If you are preparing to defend these designs in a room — a systems-design loop for an ML platform, ML data engineering, or staff data role — the companion guide turns each chapter into a drill: ML Data Architecture — Interview Prep. It assumes you have read this one.

✓ Check yourself

Answer from memory before opening each one.

1. In the traced fraud decision, the daily job recomputes yesterday's served vectors and finds 99.4% match, with the mismatch all in ip_velocity_1h. Is that a failure? What would make you act?

   Show answer

   No. The mismatch comes from late-arriving events and sits inside the 2% objective, so it is expected. Act if the rate exceeds the tolerance, or if a different feature starts to mismatch, since that points to a definition or time-semantics difference. See one fraud decision, end to end.

2. A regulator asks why an account was sent to review. Which artifacts answer, and what must have been written at decision time?

   Show answer

   The decision log returns the exact feature vector, score, model version and request ID, lineage walks each feature to its source columns, and the dataset card shows the consent policy the model trained under. The vector, model version and request ID must be logged when the request is scored. See one fraud decision, end to end.

3. Your CTO wants Flink for all features in month one. Using the roadmap, what is the order and why?

   Show answer

   Sequence by risk: reproducible training sets first, then a feature store with an online path and monitoring, and streaming last. A team that starts with the stream ends with real-time features feeding a model nobody can retrain. The exit criteria at each stage show when to move on. See crawl, walk, run.

4. The stack rejects a standalone vector database as the source of truth. What failure does that choice prevent?

   Show answer

   Embeddings that cannot be regenerated because the source text was deleted and the model version was not recorded. Keep vectors as governed lake tables with the model version, and use a small serving index as a copy. See stack choices, defended.

5. Estimate the online store size and the read rate for ranking at 2,000 requests per second with 20 features. What does that tell you about the design?

   Show answer

   About 280 thousand entities times 40 features times 80 bytes is around 1 GB, so memory is not the issue. Ranking makes about 40 thousand key reads per second, which forces pipelined reads within the 5 ms budget. See capacity and cost math.

6. Pick three anti-patterns that a team could ship with a good offline AUC, and name the check from this guide that exposes each.

   Show answer

   Training on now exposes itself in a point-in-time test that asserts feature timestamps do not exceed prediction time. A model with no baseline shows up when promotion is gated on a recorded training distribution. Deletion that stops at the tables is caught by an end-to-end erasure test across the stores, the log, the vector index and the snapshots. See anti-patterns.

Takeaway

• The plane is lakehouse-centric with a streaming lane, one registry, two stores, a snapshot-pinned builder, and lineage, monitoring, governance and orchestration wrapped around all of it.
• For any prediction the plane must say what the model saw, why it was allowed to see it, and how to see it again, and the served-feature log, lineage and dataset cards are how.
• Choose each layer for the failure it prevents, and name what you rejected and why.
• Sequence the build by risk, with reproducible datasets before a feature store, a feature store before streaming, and monitoring before all of them.
• Size the plane in order-of-magnitude numbers: the online tier is small, the streaming lane dominates run cost, and the GPU line is where optimisation belongs.
• The anti-patterns are mostly the same few mistakes seen from different sides: two definitions, training on now, unmeasured seams and deletion that stops early.

To practise defending these designs under questioning, continue with the companion drills. → ML Data Architecture: Interview Prep

Further reading

• Hidden Technical Debt in Machine Learning Systems — Sculley et al., NeurIPS 2015. The anti-patterns chapter's source argument about glue code, entanglement and unstable data dependencies.
• TFX: A TensorFlow-Based Production-Scale Machine Learning Platform — Baylor et al., KDD 2017. A reference design for an end-to-end production ML pipeline.
• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — Breck et al., Google, 2017. A checklist for scoring a plane's data, model and monitoring readiness.
• Data Validation for Machine Learning — Breck et al., SysML 2019. The validation layer of the plane as built at Google.
• Rules of Machine Learning: Best Practices for ML Engineering — Google, accessed 2026. Practical rules for the pipeline, monitoring and feedback loops.