Practice Questions, Graded

L1. Join the customer table to usage and build a CSV each week.

L2. A scheduled job that computes features per customer, joins the churn label, writes a Parquet training set with a date in the name.

L3. Defines the label precisely (churned = no activity 30 days after the prediction date); builds features as of the prediction date with point-in-time joins so no post-outcome data leaks in; versions the dataset as a tagged snapshot; records row counts, label rate and feature stats per run; a validation gate compares against the last accepted set.

L4. All of L3, plus: multiple prediction dates per customer to enlarge the set without leakage; the same feature definitions serve online scoring so the model sees in production what it saw in training; the dataset ID is written to the model registry; the pipeline is a reusable pattern for the next model. Deep dive: 02 · Training-Data Pipelines.

L1. Standardise on one SQL query and share it.

L2. Introduce a feature store; register the feature once; teams read it offline and online.

L3. A feature registry with owner, definition, freshness and tests; one materialisation job feeding an offline table and an online store; migrate the twelve models by mapping each private definition to the canonical one, measuring the delta on each model's validation before switching.

L4. Recognises this is a governance problem wearing a tooling costume: the registry needs ownership and a change process (a changed definition is a breaking change with versioning), a discovery surface so the next team finds it, and a metric on reuse. Chooses build-vs-buy by team count and real-time needs (matrix). Deep dive: 04 · The Feature Store.

L1. Query the events table at request time.

L2. A streaming job over the clickstream keeps a per-user window in Redis; the model reads it at scoring.

L3. Event-time windows with watermarks and late-event handling; idempotent updates keyed by event ID; a TTL on the window; the same aggregation logic used to backfill the offline feature for training, so offline and online agree; freshness SLO monitored end to end (event to store).

L4. Asks whether ten minutes is a requirement or a guess and proposes measuring lift at 10 minutes vs 1 hour vs daily before paying for streaming; designs the fallback when the stream lags (serve the batch feature and flag it); scopes streaming to this feature only. Deep dive: 06 · Streaming Features.

L1. "A lot; use a vector database."

L2. 50M × 768 dims × 4 bytes ≈ 150 GB of vectors; an HNSW index roughly doubles it; needs a cluster with ~300 GB of memory.

L3. Same arithmetic, then the levers: int8 quantisation to ~40 GB, product quantisation to ~10 GB with recall loss you'd measure; IVF or DiskANN for disk-resident serving if QPS allows; embedding compute is the real cost — 50M images at ~200 images/s/GPU is ~70 GPU-hours per full re-embed, so the model version cadence is a budget decision.

L4. Separates offline (store all vectors in the lakehouse, cheap) from online (index only what's servable — in-stock items, maybe 20%); reasons about rebuild strategy on model upgrade and how to A/B two embedding versions; notes egress if images live in a different cloud than the GPUs. Deep dive: 12 · Cost & Scale.

L1. Put it in S3 and share the bucket.

L2. Object storage, Parquet, partitioned by device and date; a catalog so teams can find it; labels in a separate table joined by key.

L3. Iceberg tables over object storage for the tabular parts with snapshot tagging so each experiment pins a version; a training-set manifest per team rather than copies; tiering (hot recent months, cold archive) with lifecycle rules; data loading designed for GPU throughput — sharded files sized 100 MB–1 GB, locality with the training cluster, no small-file explosion.

L4. Labels as a versioned, auditable dataset of their own (who labelled, when, with which policy) because five teams will disagree about them; a branching layer (LakeFS) if teams need to experiment on the corpus without copying 200 TB; access controls and a cost-attribution model per team so the shared corpus doesn't become a tragedy of the commons. Deep dive: 05 · Storage, Formats & Versioning.

L1. A vague story about a model performing worse in production; fixed by retraining.

L2. A concrete feature (a null-default on timeout online, a value present offline); found by comparing distributions; fixed the online path.

L3. Found it because features-as-scored were logged and joined back to an offline recomputation for the same entities; fixed the specific feature and made the skew check a standing monitor; quantified the model impact.

L4. Changed the system so the failure class can't recur: one feature definition materialised to both stores, the skew monitor as a platform default for every model, and a post-incident review that named the missing gate. Talks about what it cost and what it saved. Deep dive: 10 · Data Quality & Drift.

L1. Add more cache nodes.

L2. Check whether the 40 ms is the store or the network; look for large values or too many round trips per request; batch the reads.

L3. Decompose the request: N features across M keys → M round trips? Pipeline or multi-get to one; check value sizes (a 200-element list per key is slow to deserialise); look at hot keys and uneven partitions; confirm whether p99 spikes correlate with the batch materialisation write (write amplification during bulk loads is the usual culprit) and throttle or schedule it.

L4. Frames the SLO as a budget across embed, retrieve, model; considers precomputing the assembled feature vector per entity as one value; adds request-level tracing so the next incident is a query, not a hunt; asks whether the 10 ms SLO is real or inherited. Deep dive: 08 · Serving Architectures.

L1. Rerun the model on the customer's current data.

L2. Look up the scoring log for that request; rerun v14 on the logged inputs.

L3. Scoring log holds model version, feature values, feature timestamps and request ID; the model registry links v14 to its training-dataset snapshot tag and code commit; both are retained for the regulatory window; you reproduce the score bit-for-bit and can also show what data trained the model.

L4. Treats this as a designed capability with a retention policy, access control on the logs (they contain PII), and a documented procedure for legal; discusses what to do when the feature store's history was compacted — which is why point-in-time retention is a governance decision, not a storage optimisation. Deep dive: 09 · Lineage & Reproducibility, 11 · Governance & Privacy.

L1. Delete their rows from the database.

L2. Delete from source tables and from the feature store; the lakehouse needs a delete that rewrites files (Iceberg row-level deletes); training sets get rebuilt.

L3. An erasure pipeline keyed by customer ID that fans out to every table with lineage; snapshot expiry so old versions don't retain the rows beyond the window; training sets identified via lineage and either regenerated or documented as pseudonymised; a record of completion.

L4. The model question honestly: retraining on demand is impractical, so the policy is either scheduled retrains within the SLA window, pseudonymisation at ingestion so training data never holds direct identifiers, or a documented legal basis — a decision made with counsel, encoded in the platform, and tested. Deep dive: 11 · Governance & Privacy.

L1. COUNT(*) grouped by customer over the last 30 days from today.

L2. A join from prediction dates to orders with order_ts < prediction_ts and a 30-day lower bound, grouped by customer and prediction date.

L3. Same, with the boundary semantics stated (strictly before the prediction timestamp, not the date), handling customers with zero orders via a left join and COALESCE, and awareness that the naive join fans out and must be aggregated before joining if the orders table is large.

SELECT p.customer_id,
       p.prediction_ts,
       COALESCE(COUNT(o.order_id), 0) AS orders_30d
FROM   prediction_points p
LEFT JOIN orders o
  ON   o.customer_id = p.customer_id
 AND   o.order_ts <  p.prediction_ts                       -- strictly before: no leakage
 AND   o.order_ts >= p.prediction_ts - INTERVAL '30 days'
GROUP  BY p.customer_id, p.prediction_ts;

L4. Notes that order_ts should be the time the order was known to the system (arrival/ingest time), not the business time, if late-arriving orders exist — otherwise the feature uses information that wasn't available at prediction time; mentions ASOF joins where the engine supports them and that the feature store generalises exactly this pattern. Deep dive: 02 · Training-Data Pipelines.

L1. The model is great.

L2. Look for a feature that's a proxy for the label (a "refund_flag" in a fraud set); drop it.

L3. Systematic: rank features by single-feature AUC; check timestamps of each feature's source against the prediction time; check the split — random splits leak across time and across the same entity; check for duplicated rows across train and validation; check labels computed from data that also feeds features.

L4. Adds the automated version to the dataset validation gate — single-feature AUC ceiling, timestamp audit, entity-grouped time-based splits — so the next leak is caught before a human looks. Deep dive: 02 · Training-Data Pipelines.

L1. On each change, re-embed the document and upsert.

L2. Consume the change stream; for updates re-chunk and re-embed only if the content hash changed; upsert chunks by (doc_id, chunk_idx); on delete remove all chunks for the doc.

L3. Adds: micro-batching to use the embedding model efficiently; idempotency keyed by (doc_id, content_hash, model_version) so replays are harmless; handling the chunk-count change on edit (delete chunks beyond the new count); a dead-letter path for parse failures; lag and backlog metrics with an alert.

L4. Treats the chunk table as the source of truth and the index as derived, so the job writes the table and a separate process syncs the index — which makes a full rebuild on model upgrade the same code path as the incremental one. Deep dive: 07 · Embeddings & Vector Data.

L1. "They wanted real-time and I said batch was fine." No reasoning, no outcome.

L2. A specific request (streaming features for a model), a cost argument, and an agreement to start with batch.

L3. Reframed the disagreement as a measurable question — what lift does freshness buy — ran the experiment (train with 1-hour vs 1-day-old features), let the number decide, and documented the flip condition for revisiting.

L4. Turned the outcome into a platform norm: freshness tiers with costs attached, so the next request comes with a lift estimate instead of a demand; kept the relationship intact by giving the team a way to win the argument with evidence.

L1. Whatever the loudest team asks for.

L2. A backlog prioritised by number of requesting teams and effort.

L3. Instrumented pain: time-to-first-model, feature reuse rate, incidents per model, cost per training run — and picks the work that moves the worst metric for the most teams; pairs each platform capability with a lighthouse model that proves it.

L4. Adds the strategic layer: what the business will need models for in 12 months (real-time? multimodal? regulated?) and builds the data foundations ahead of the first request, because feature stores and lineage take quarters to land and days to demand; says no to bespoke work with a reason and an alternative.