The Method: Five Moves for Any ML Data Design Question

Why a method

ML data design prompts are open: "design the data platform for our recommendation system." Candidates fail in two symmetrical ways — freezing, because there is no obvious first line to draw, or rambling, because there are too many. A fixed sequence of moves solves both. It also matches how the graders think: they are checking that you covered both paths (training and serving), that you handled time correctly, and that you can run it. The five moves below are ordered so that each one produces the inputs the next one needs, and so that the two things most candidates forget — labels and operations — have a reserved slot.

Run the moves in roughly these proportions in a 45-minute round: requirements and signal 6–8 min, sources and labels 5 min, offline path 10–12 min, online path 8–10 min, operate 5–7 min, with the remainder for the interviewer's deep dive. If the round is 60 minutes, the deep dive grows; the moves don't.

Move 1 · Requirements & the signal

Before drawing, establish what the model does and what that demands of the data. The word to use is signal: which decision does a prediction drive, and what does that decision need from us?

• The decision. Block this payment? Rank these 200 listings? Forecast next week's capacity? The decision fixes the latency budget and the cost of being wrong.
• Latency of a prediction. Synchronous in the request path (tens of ms), near-real-time (seconds), or batch (nightly scores written to a table)? This single answer decides whether an online store exists at all.
• Freshness of the inputs. How stale can a feature be before the prediction degrades? "Transactions in the last five minutes" is a streaming feature; "average order value over 90 days" is a nightly batch one. Most systems have both.
• Volume. Predictions per second, entities (users, items, accounts), features per entity, event rate feeding features. Get orders of magnitude, not precision.
• Labels. What is ground truth and when does it arrive? (Move 2 goes deep; here you just flag the delay.)
• Correctness bar and constraints. Regulated? Explainability required (which forces feature logging)? PII in the inputs? Budget and team size?

Move 2 · Sources & labels

List the raw data and, separately, the ground truth. Candidates who skip labels design a beautiful pipeline that cannot evaluate or retrain its own model.

• Sources. Transactional tables (via CDC), event streams (clicks, transactions, telemetry), third-party enrichments, documents or media for embedding. For each: owner, rate, arrival pattern, and whether it changes after the fact (late events, corrections, deletes).
• Labels. Where does the outcome come from — a chargeback system, a "purchased" event, a human review queue, a future value of a metric? What is the label delay (seconds for clicks, weeks for fraud, months for churn)? Is the label noisy or partial (only reviewed cases get labelled)? Does it get revised?
• The join key and time. How does a prediction get joined back to its label — by request id, by entity and window? This join is the spine of the training set and of every evaluation.

Say the label delay out loud and draw it as a number on the board ("labels arrive T+30d"). It drives how far back your training window must start, how you evaluate a freshly deployed model, and whether you can retrain weekly at all. Depth: Deep dive 03 · Labels, Ground Truth & Feedback Loops.

Move 3 · The offline path

The path from raw data to a training set. Draw it left to right: raw → curated entity tables → feature computation → offline feature store (or feature tables) → point-in-time training set → versioned dataset → training job.

• Feature definitions. One definition per feature, owned, versioned, with an entity key and an event-time semantics ("sum of transaction amounts in the trailing 7 days, as of event time"). This is the artifact both paths share.
• Point-in-time correctness. Every feature value in the training set must be the value that was knowable at the prediction time of that row. That means an as-of join between the label/prediction timestamps and the feature history — never a join to "current" values. Draw the as-of join explicitly; it is the single most tested mechanism in these loops.
• Backfills. A new feature needs history. Say how you compute it for the past (from raw events, idempotently, by partition) and how you keep backfilled history distinct from live values.
• Dataset versioning. The training set is an immutable, addressable artifact — a table snapshot, a manifest of file hashes, a tagged commit in a versioned lake. The model registry points at it.
• Splits and leakage. Time-based splits for anything with drift; entity-disjoint splits when entities repeat; leakage checks for features that encode the label.

Depth: Deep dive 02, 04, 05.

Move 4 · The online path

Only if Move 1 said predictions happen at request time. The path from live events to the features a model reads in milliseconds.

• Online store. A low-latency key-value view of the latest feature values per entity — Redis, DynamoDB, Cassandra, or a purpose-built store. Sized by entities × features × bytes, with a TTL.
• Materialisation. Batch features are pushed from the offline store on a schedule; streaming features are computed by a stream processor (Flink, Spark Structured Streaming, a Kafka Streams app) and written directly. Both write the same feature names with the same semantics.
• Request-time assembly. The service looks up features by entity key, adds request-only features (the current basket, the query), and calls the model. State the latency budget: total, and the slice feature retrieval gets.
• Feature logging. Log the exact feature vector the model saw, keyed by request id. This is how you build the next training set without recomputation, how you debug a prediction, and how you measure skew.
• Skew. The offline value for (entity, time) and the online value the model saw should agree. Name the mechanism that keeps them agreeing (one definition compiled to both paths, or the same code) and the reconciliation job that checks it.

Depth: Deep dive 06, 08.

Move 5 · Operate

The move most candidates never reach and the one that most reliably marks a senior. Five words on the board: quality, drift, lineage, cost, governance.

• Quality gates. Schema and distribution checks on features before they reach training or the online store; a bad batch is quarantined, not served.
• Drift and skew monitoring. Feature distributions over time versus the training baseline; online-vs-offline reconciliation; prediction distribution; label-arrival lag.
• Lineage and reproducibility. Model version → dataset version → feature definitions → raw partitions and code commit. "Can we rebuild model v14 from scratch?"
• Cost. Storage of feature history and training sets, compute for backfills, the online store's memory, egress to training clusters. Name the biggest line and how you'd cap it.
• Governance. PII in training data, consent and retention, the right to erasure propagating into datasets already trained on, access control on features, audit for regulated decisions.

Depth: Deep dive 09, 10, 11, 12.

Estimation cheat sheet

Numbers you should be able to produce in under a minute. Precision is irrelevant; the order of magnitude decides the architecture.

The board layout

Draw two horizontal lanes that share a spine. The spine is the feature definitions; the lanes are the offline and online paths; operations sit underneath as a band. Every drill in this guide uses this layout.

  SOURCES                 OFFLINE PATH (training)
  ┌──────────┐   CDC   ┌────────────┐  ┌──────────────┐  ┌────────────────┐  ┌─────────┐
  │ OLTP DBs │───────▶ │ raw / lake │─▶│ feature jobs │─▶│ offline store  │─▶│ as-of   │──▶ dataset vN ──▶ train
  ├──────────┤ stream  │ (Iceberg)  │  │ (batch)      │  │ (history, PIT) │  │ join    │        │
  │ events   │───┬───▶ └────────────┘  └──────────────┘  └───────┬────────┘  └────▲────┘   model registry
  ├──────────┤   │                                    materialise │              │  labels (T+delay)
  │ labels   │───┼─────────────────────────────────────────────── ┼ ─────────────┘
  └──────────┘   │      ONLINE PATH (serving)                      ▼
                 │     ┌──────────────┐    ┌──────────────┐   ┌────────┐   ┌───────────┐
                 └───▶ │ stream jobs  │───▶│ online store │◀──│ model  │◀──│ request   │
                       │ (Flink)      │    │ (Redis/KV)   │   │ service│   │ + logging │
                       └──────────────┘    └──────────────┘   └───┬────┘   └───────────┘
                                                                  └──▶ feature log ──▶ back to offline
  ────────────────────────────────────────────────────────────────────────────────────────
   OPERATE:  quality gates ◆   drift & skew ◆   lineage ◆   cost ◆   governance/PII ◆
  ────────────────────────────────────────────────────────────────────────────────────────
   SPINE: feature definitions (name · entity · event-time semantics · owner · version)

Mark the decision points with a ◆ as you go — batch vs streaming for each feature group, buy vs build the store, warehouse vs lakehouse for history, which online store. Those marks are your invitations for the deep dive; you are choosing where the interviewer digs.

Narrating it

The method is also a script. A strong opening sounds like this:

Two habits carry the round. First, the decision sentence at every fork: "Given [requirement], I'd choose [option] over [alternative] because [fit]; the cost is [downside], and I'd revisit if [condition]." Second, hand over the wheel at the end of each move: "That's the offline path — the interesting decisions were the as-of join and the versioning. Want to go deeper there, or shall I draw serving?" Driving the structure while letting the interviewer pick the depth is what a senior discussion feels like from their side of the table.

The method at four levels

Now the method applied to the question you are most likely to get: 03 · Drill: Design a Feature Store.

✓ Check yourself

Answer from memory before opening each one.

1. Product wants a churn score for 2M customers, written to a table every night. What do Move 1's answers tell you about an online store and streaming features?

   Show answer

   Neither is needed. Prediction latency is the answer that decides whether an online store exists, and a nightly score is a warehouse job writing a table. Adding Redis and Flink here would be cost with no requirement behind it. See Move 1 and the estimation sheet.

2. A card-payments model handles 8M transactions a day with a peak factor of 5, over 10M cards and 120 features per card. Estimate peak predictions per second and the online store size, then say what the numbers imply.

   Show answer

   8M / 86,400 is about 93 a second on average, so roughly 460 at peak. The store is 10M x 120 x 8 B x 2.5 overhead, about 24 GB. That is a single-node or small-cluster in-memory problem, so size alone does not force a distributed store. See the estimation sheet.

3. Why is joining labels to the "current" value of a feature wrong even when the feature table is correct today, and what does the as-of join compare instead?

   Show answer

   The current value reflects things that happened after the prediction, so the model trains on information it could not have had. The as-of join compares each row's prediction timestamp with the feature history and takes the newest value knowable at that time. See Move 3.

4. Chargebacks arrive about 60 days after a transaction. What does that do to your training window, your evaluation of a freshly deployed model, and a weekly retrain?

   Show answer

   The freshest fully labelled data is about two months old, so training windows must end then. A newly deployed model cannot be judged on true labels for about two months, so you need proxies in the meantime. A weekly retrain is still possible, but each one trains on data that is already about two months stale. See Move 2.

5. Why log the exact feature vector the model saw at scoring time, and what second mechanism do you pair with a shared feature definition?

   Show answer

   The log gives you the next training set without recomputation, a way to debug any single prediction, and the raw material to measure skew. Pair the shared definition with a reconciliation job that recomputes offline values and compares them with the logged ones, because a shared definition alone does not prove agreement. See Move 4.

Takeaway

• Every ML data design question can be run as five moves: requirements and signal, sources and labels, the offline path, the online path, and operations.
• Five questions fix the shape of the system: the decision, whether it is made in the request path, input freshness, volume, and when the label arrives.
• Prediction latency decides whether an online store and streaming features exist at all. A nightly score needs neither.
• Every training row must carry the feature values knowable at its own prediction time, through an as-of join to feature history, never current values.
• Log the vector the model saw, share one definition between both paths, and run a reconciliation job to catch skew.
• Estimates decide forks. State the number, then its consequence for the design.

With the method fixed, apply it to the question you are most likely to get. → Drill: Design a Feature Store

Further reading

• Hidden Technical Debt in Machine Learning Systems — Sculley et al., NeurIPS, 2015. The source for why data dependencies, glue code and feedback loops, not models, dominate ML system cost.
• Rules of Machine Learning — Google for Developers. Practical rules on training-serving skew, feature logging and pipeline ownership.
• Point-in-time joins — Feast documentation. How an as-of join over feature history prevents leakage in a training set.
• Timely Stream Processing — Apache Flink documentation. Event time versus processing time, the clock behind every streaming feature.