The Big Trade-offs

How to use a matrix in the room

A trade-off matrix is not something you recite; it's the shape of your reasoning when the interviewer says "why not the other one?" The pattern that works: name the requirement that dominates, pick, state the cost honestly, and name the flip condition. The rows below are the dimensions interviewers actually pull on. Learn the flip conditions cold — they are the difference between a defended decision and a preference.

Batch vs streaming features

The most common fork, and the one where candidates over-reach. The default is batch; streaming earns its place when a specific feature's value decays in minutes. Deep dive: 06 · Streaming Features.

Warehouse vs lakehouse for ML

A warehouse is a superb place to compute tabular features; it is a poor place to store images, tensors, or ten thousand versioned training sets. The fork is really "where does the ML data live", not "do we keep Snowflake". Deep dive: 05 · Storage, Formats & Versioning and 13 · Reference Architectures.

Build vs buy a feature store

The honest starting point: most teams need the discipline of a feature store (registry, point-in-time correctness, one definition offline and online) before they need the product. Deep dive: 04 · The Feature Store.

Online store choices

The online store serves a feature vector for one entity in single-digit milliseconds. The choice is dominated by access pattern (point reads by key), write pattern (batch materialisation vs streaming updates), and how much operational burden you can carry. Deep dive: 08 · Serving Architectures.

Dataset versioning approaches

"Which data trained model v14?" must be answerable in one query, for reproducibility, audit and debugging. The mechanisms differ in granularity and in what they version — table snapshots, files, or a whole repository of data. Deep dive: 09 · Lineage & Reproducibility.

Vector store choices

Size the problem before naming a product: vectors × dimensions × bytes. Under a few million vectors an extension in the database you already run is usually right. Deep dive: 07 · Embeddings & Vector Data.

The decision sentence

Every matrix above collapses to one sentence in the room. Practise it until it's reflex:

The flip condition is the senior tell. It shows you know the boundary of the decision, not just the decision — and it hands the interviewer a natural follow-up you're already prepared for.

Now put the matrices under pressure: 09 · Practice Questions, Graded.

✓ Check yourself

Answer from memory before opening each one.

1. A product manager wants a 28-day spend feature "real-time". How do you decide, and what does the answer say about batch versus streaming failure modes?

   Show answer

   Default to batch. A 28-day aggregate barely moves in minutes, so streaming costs an always-on cluster for no lift, and you stream only when measured lift from freshness pays for the operations. Batch fails by going stale, which is visible. Streaming fails by being quietly wrong, because watermarks, late events and exactly-once are yours to get right. See batch versus streaming.

2. Warehouse backfill credits dominate the bill, training jobs need exports, and the corpus now includes 30 TB of images. Where does each kind of data live?

   Show answer

   Compute tabular features where the governed data already is, and land training sets and unstructured data in the lakehouse where training jobs read directly. If backfill credits or export volume keep dominating, move to lakehouse-only. See warehouse versus lakehouse.

3. Three models, one team, nightly batch. A vendor offers a managed feature store. What do you do, and what event changes your answer?

   Show answer

   Build the discipline, not the product: versioned feature tables, a registry in code, a Redis cache if anything is online, and strict point-in-time joins. Buy when a second team needs the same features online, or when a real-time feature's freshness SLO becomes a contract. See build versus buy.

4. The entity space is 300M, a nightly backfill rewrites 200M rows, and p99 reads must be under 10 ms. What does each online store cost you, and what decides it?

   Show answer

   Redis is memory-bound at tens of GB per node, so a hot set that large gets expensive. DynamoDB handles the size but its write bill for a 200M-row backfill can become the largest line. Bigtable or Cassandra with bulk loading suit that write pattern. The decision is size times write pattern. See online store choices.

5. A training set spans three tables and a folder of images and must be reproducible as one unit. Why is an Iceberg snapshot tag not enough?

   Show answer

   Iceberg versions one table at a time, with no atomic commit across several tables and loose files. LakeFS gives an atomic commit and branches across a whole repository, so the tables and images are consistent as a unit. See dataset versioning approaches.

6. You have 2M vectors, already run Postgres, need joins to orders, and expect about 20 QPS. Which store, and what would make you move?

   Show answer

   pgvector, because at that size an extension in the database you already run is usually right and joins are easy. Move to a dedicated engine if vectors grow past a few million or filtered search at higher QPS starts to hurt recall. Size the problem before naming a product. See vector store choices.

Takeaway

• Default features to batch. Stream only those whose value decays in minutes, and only when measured lift justifies the operating cost.
• Batch fails stale and streaming fails silently wrong, and that difference sets how you monitor each.
• Compute tabular features where governed data already lives, and put training sets and unstructured data in the lakehouse.
• Below about five models, build the feature-store discipline. Buy when a second team needs the features online or freshness becomes a contract.
• An online store is chosen by entity count times write pattern: Redis in memory, DynamoDB or Bigtable at very large size, and bulk-load cost flips the choice.
• Tag an Iceberg snapshot for single tables, and add LakeFS for multi-table or file-backed datasets or for branches. Size vectors before you name a vector store.
• Every fork is a decision sentence with a cost and a flip condition.

Next, put those matrices under pressure with questions graded from L1 to L4. → Practice Questions, Graded

Further reading

• Feast documentation — Feast. The open-source feature store whose registry and stores anchor the build-versus-buy row.
• Tecton documentation — Tecton. The managed side of the same fork, with streaming features and online serving included.
• Branching and Tagging — Apache Iceberg documentation. Tags and branches on table snapshots for dataset versioning.
• Table batch reads and writes — Delta Lake documentation. Includes time travel and its retention behaviour.
• lakeFS documentation — lakeFS. Git-style branches and atomic commits across object-storage data.
• DVC documentation — Data Version Control. Versioning files and directories alongside code in git.
• pgvector — pgvector project README. Vector similarity search inside Postgres, for the small-corpus end of the vector fork.