Drill: Design a Feature Store

The prompt

Notice what the prompt contains: multiple models, multiple teams, an explicit pain (mismatch), and no numbers. Your first job is to convert it into requirements; your second is to resist saying "Feast" for at least ten minutes.

Clarifying questions

Ask these, in roughly this order, and write the answers on the board as you get them.

• Which predictions are online? "Ranking and recommendations are in the request path; seller risk runs nightly." → Two models need an online store; one doesn't.
• Latency budget? "Search has 100 ms end to end." → Feature retrieval gets a slice, say ≤ 15 ms p99.
• Freshness? "Ranking wants the last few clicks; most other features can be a day old." → A small set of streaming features, a large set of batch features.
• Scale? "10M active buyers, 500k sellers, 50M listings; ~2,000 search requests/s peak." → Sizes the online store and the retrieval fan-out.
• Who defines features today, and who should? "Each team; we want sharing." → Feature definitions are a shared, owned artifact — a registry with ownership is a requirement, not a nicety.
• What does the training side look like? "Nightly retrains from the warehouse, weekly for risk." → The offline store lives beside the warehouse; point-in-time retrieval is needed.
• Where is data today? "Snowflake plus Kafka for events." → Build the offline store on the warehouse; streaming features from Kafka.

Move 1 · Requirements & estimate

Say the consequence column out loud. It is the argument for everything you draw next.

Move 2 · Sources & labels

Sources: the listings and orders databases via CDC (Postgres → Debezium → Kafka), clickstream and search events from the web/mobile clients (Kafka), and a seller-verification service. Labels: for ranking, clicks and purchases joined to the search request id (delay seconds to hours); for recommendations, the same; for seller risk, chargebacks and policy actions (delay ~14 days, revised on appeal). Draw the label delay next to each model. The feature store does not own labels, but the training-set builder that sits on top of it does the join, so the entity keys and timestamps must line up.

The board

                     ┌─────────────────────────────────────────────────────────┐
                     │  FEATURE REGISTRY  (definitions · entity · owner ·      │  ◆ build vs buy
                     │  event-time semantics · version · SLA)  ── the spine    │
                     └───────────────┬───────────────────────┬─────────────────┘
        batch definitions ───────────┘                       └───────── streaming definitions
                 │                                                             │
  Snowflake ──▶ ┌▼─────────────────┐  materialise (hourly/daily)  ┌────────────▼──────────┐
  (CDC'd tables)│ OFFLINE STORE    │ ───────────────────────────▶ │ ONLINE STORE (Redis)  │◀── Flink jobs ◀── Kafka
                │ feature history  │                              │ latest value / entity │     (clicks, views)
                │ (Iceberg/Snowflk)│                              │ TTL · multi-get       │
                └┬─────────────────┘                              └────────────┬──────────┘
                 │ as-of join with labels                               get_online_features(entity keys)
                 ▼                                                             │
          ┌──────────────┐   dataset vN     ┌────────────┐          ┌──────────▼─────────┐
          │ training-set │ ───────────────▶ │ registry / │          │ ranking / recs     │──▶ response
          │ builder (PIT)│                  │ model vN   │          │ service            │
          └──────────────┘                  └────────────┘          └──────────┬─────────┘
                 ▲                                                             │ feature log (request_id, vector)
                 └─────────────────────────────────────────────────────────────┘
  ────────────────────────────────────────────────────────────────────────────────────────
   OPERATE: schema+distribution gates ◆  offline↔online reconciliation ◆  freshness SLOs ◆  lineage ◆  cost

The components, and what each is for

Weak answers list the boxes; strong answers say the problem each box exists to solve.

• Registry. The shared definition of every feature: name, entity, source, transformation, event-time semantics, owner, version, freshness SLA. It exists so that "buyer_7d_purchases" means one thing in the ranking model and the risk model, and so that two teams don't compute it twice. It is also the thing that compiles to both paths — the same definition drives the batch job and the stream job. Without it a feature store is just two databases.
• Offline store. Full feature history — every value with the timestamp it became true — in the warehouse or lakehouse. Exists for point-in-time training sets and backfills. Big, cheap, columnar.
• Online store. The latest value per entity, in memory, with a TTL. Exists for the request path. Small, fast, expensive per byte. It holds only the features online models read — a subset of the registry.
• Materialisation. The scheduled push of batch features from offline to online, and the stream jobs writing fresh features straight to online. Exists so the model reads yesterday's batch values and this second's streaming values through one interface.
• Training-set builder. Takes a labelled "spine" (entity, timestamp, label) and joins features as of each timestamp. Exists to prevent leakage. Output is a versioned dataset artifact.
• Serving SDK / API. get_online_features(entity_keys, feature_list) with batched multi-get, and get_historical_features(spine, feature_list) for training. Exists so consumers never touch the stores directly.
• Feature logging. The vector the model actually saw, keyed by request id. Exists for skew measurement, debugging, and cheap training-set construction.

Point-in-time retrieval

This is where the round is won. The training-set builder must, for each labelled row at time t, fetch the feature value that was current at t — not the latest, not the value at the end of the day. In SQL that is an as-of join against the feature history:

-- spine: one row per (buyer_id, search_ts, label)
-- fh:    feature history, one row per (buyer_id, feature_ts) with values valid from feature_ts
SELECT s.buyer_id, s.search_ts, s.label,
       f.buyer_7d_purchases, f.buyer_30d_gmv
FROM spine s
ASOF JOIN feature_history f
  MATCH_CONDITION (s.search_ts >= f.feature_ts)     -- the latest value at or before the search
  ON s.buyer_id = f.buyer_id;

-- engines without ASOF: window over the union, or a lateral "latest row where ts <= t"
-- Either way the invariant is the same: feature_ts <= search_ts, and strictly the newest such row.

Two refinements mark a senior answer. First, a maximum staleness: a value from 40 days ago should not be "the latest" for a 7-day feature — enforce search_ts − feature_ts ≤ ttl and treat the rest as null. Second, event time vs availability time: if a feature computed at 02:00 describes yesterday, its value was not knowable until 02:00, so the as-of comparison must use the time it became available, or the training set will contain values the model could not have had. Depth: Deep dive 04.

Trade-offs

What the interviewer is listening for

• That you separated the registry (the product) from the stores (the implementation).
• The as-of join, unprompted, with the staleness bound and the availability-time subtlety.
• An online-store size and a latency budget with numbers.
• Only the features that need it are streamed; the rest are batch. No cargo-cult Flink.
• Feature logging at serving time, and a reconciliation job that turns "training-serving mismatch" from an anecdote into a metric.
• A build-vs-buy decision with a flip condition, and an honest "you might not need this yet" for the nightly risk model.

Follow-up probes

• "A team adds a new feature and needs two years of history. What happens?" — A backfill: run the definition over raw history partition by partition, idempotently, writing to the offline store with the correct event timestamps; flag the rows as backfilled; do not push to online until the live job is producing. Cost is the raw scan; cap it by computing from curated entity tables where possible.
• "How do you know online and offline agree?" — Log the served vector by request id; nightly, recompute the same features offline as-of the request time and diff; alert on drift above a per-feature threshold. Root causes are almost always time-window edges or a UDF that diverged between engines.
• "Redis goes down." — The model service degrades: default values or last-known from a local cache for batch features, skip streaming features, and log that it did. Ranking gets worse, not broken. Rehydrate from the offline store's latest snapshot; streaming features catch up from Kafka offsets.
• "Two teams want different definitions of buyer_7d_purchases." — They get two names, or one versioned definition; the registry enforces uniqueness and ownership. This is a governance question wearing a technical hat.
• "Why not just query Snowflake at request time?" — Latency and cost: warehouse queries are hundreds of ms to seconds and priced per scan; 2,500 req/s would be ruinous. The online store is a cache with a contract.

The answer at four levels

Next, the mechanism this drill leaned on, as its own question: 04 · Drill: Training Data with Point-in-Time Correctness.

✓ Check yourself

Answer from memory before opening each one.

1. A colleague says a feature store is Redis plus Snowflake with a sync job. What is missing, and why does it matter for the mismatch problem in the prompt?

   Show answer

   The registry is missing: one owned, versioned definition per feature that drives both the batch and the stream path. Without it you have two databases and two teams free to compute the same name differently, which is the original problem. See the components.

2. A feature is computed by a job that runs at 02:00 and describes the previous day. Training joins it on the date it describes. What goes wrong?

   Show answer

   Rows between midnight and 02:00 get a value the system did not have yet, so the training set contains information production could never serve. The as-of comparison must use the time the value became available, not the time it describes. See point-in-time retrieval.

3. A buyer's last feature row for buyer_7d_purchases is 40 days old. What does a plain as-of join return, what should it return, and why?

   Show answer

   A plain as-of join returns the 40-day-old row as "the latest". It should return null, because a 7-day feature that old would have expired from the online store. Enforce that search time minus feature time is within the TTL so training sees what serving would. See point-in-time retrieval.

4. The online store loses all its data at 14:00. Which features recover first, and how does each kind come back?

   Show answer

   Batch features come back by re-materialising the latest snapshot from the offline store. Streaming features rebuild as the stream jobs resume from their Kafka offsets. Until then the service uses defaults or a local cache, flags the degraded state in its log, and keeps serving. See the board.

5. Search and recommendations run online, and seller risk is a nightly batch model. Do you put seller risk on the feature store? State what decides it.

   Show answer

   Put its definitions in the registry so they are shared and owned, and keep the batch features as warehouse tables with point-in-time joins. It needs no online store. The answer changes if the model moves into the request path, since then it needs materialisation. See the trade-offs.

6. The nightly reconciliation shows the p95 difference for buyer_7d_purchases rising from near zero to 8% after a deploy. What are the likely causes and what do you check first?

   Show answer

   The usual causes are a time-window edge that differs between engines, such as an inclusive versus exclusive boundary, or a UDF that diverged between the batch and stream implementations. Check the compiled definitions and the window boundary semantics first. The reconciliation turned a vague mismatch into one named feature. See the follow-up probes.

Takeaway

• A feature store is a registry, two stores and the code that keeps them consistent. The registry is the product and the stores are implementation.
• The offline store holds full history for point-in-time training sets. The online store holds the latest value per entity in memory with a TTL, and only for features that online models read.
• The training-set builder joins features as of each row's time, uses availability time rather than event time, and bounds staleness with a TTL.
• Stream only the features whose freshness need is shorter than the batch cadence, and batch the rest.
• Log the vector served and reconcile it nightly against an offline recompute. That turns "training-serving mismatch" into a metric.
• Build versus buy turns on team count and streaming needs. State the flip condition, and say when a nightly model does not need the platform yet.

The as-of join carried most of this drill, so the next chapter makes it the whole question. → Drill: Training Data with Point-in-Time Correctness

Further reading

• Feast architecture overview — Feast documentation. The registry, offline store, online store and materialisation as one open-source design.
• Point-in-time joins — Feast documentation. How historical retrieval avoids leakage and what the timestamp semantics are.
• ASOF JOIN — Snowflake documentation. The MATCH_CONDITION syntax used in the chapter's as-of query.
• Tecton documentation — Tecton. A managed feature platform with first-class streaming features, for the buy side of the build-versus-buy fork.
• MGET — Redis documentation. Fetching many keys in one round trip, the call behind a 15 ms retrieval budget.