Reference Architectures Compared

Three ways to build the same plane

The Data Platform Systems Design guide compares three analytics stacks — open-source lakehouse (A), Databricks-centric (B), Snowflake-centric (C). The ML data plane sits on top of one of those, but its own architectural fork is different and deserves its own names, because two companies on the same warehouse can build completely different ML planes. The fork is where the centre of gravity for ML data lives:

• W — Warehouse-centric. The cloud warehouse is the system of record for features and training data. Feature views are warehouse tables; training sets are queries; the feature store is either the warehouse vendor's (Snowflake Feature Store, BigQuery ML) or Feast with the warehouse as its offline store. Online serving is a thin materialization to a KV store. Maps naturally onto Stack C.
• L — Lakehouse-centric. Open table formats on object storage are the system of record; Spark (or Trino/Ray) computes features; the feature store is Databricks Feature Store on Unity Catalog or Feast on Iceberg/Delta; MLflow (or the equivalent) ties datasets to models; training reads sharded files directly from the lake. Maps onto Stacks A and B.
• S — Streaming-first. The event log is the system of record; features are computed continuously by Flink (or Spark Structured Streaming / RisingWave) and written to both an online store and a lake for history; training sets are built by replaying the log or reading the streamed history. The warehouse is a consumer, not the centre. Sits beside any analytics stack.

W · Warehouse-centric

Choose W when the company already runs on Snowflake or BigQuery, the models are tabular, the freshness bar is hourly-to-daily, and the ML team is small enough that "SQL is the feature language" is a strength rather than a ceiling. Feature pipelines are dbt models; point-in-time joins are ASOF JOINs; the training-set builder writes a snapshot to a stage or directly to Parquet in the lake for the trainer to read.

Strengths: one governance model (the warehouse's), one skill set, fastest time to a first model, excellent for batch scoring, and every feature is instantly available to analysts. Weaknesses: the warehouse is a poor dataloader and a poor streaming engine, per-query pricing punishes iterative experimentation, sub-minute features need a bolt-on stream path that breaks the "one system" story, and unstructured data (images, text at volume, embeddings) is a second-class citizen.

L · Lakehouse-centric

Choose L when the data plane must serve both analytics and ML at scale, when datasets include unstructured or very large tabular data, when reproducibility matters (snapshot-pinned training sets are native), and when the team can run Spark or equivalent. It is the most general shape and the one most large ML organisations converge on.

Strengths: open formats mean no egress tax between engines — Spark computes features, Trino serves analysts, Ray or PyTorch reads shards directly; time travel gives free dataset versioning; storage is cheap and the same bytes serve everyone; embeddings and vectors live beside tabular data (Lance, Iceberg with vector extensions). Weaknesses: more moving parts (catalog, compaction, snapshot expiry, engine versions); real-time features still need a separate streaming path; the "managed" flavour (Databricks) reduces the operations burden at the price of vendor gravity.

S · Streaming-first

Choose S when the product's value depends on seconds — fraud at payment time, live pricing, real-time ranking on session context — and when the team can operate Kafka and Flink in production. Every feature is a continuously maintained aggregate; the online store is always fresh because it is written by the same job that writes the history; training sets are built from the same computed history, so offline/online parity is structural rather than checked.

Strengths: the lowest possible feature latency, parity by construction, replayable history (rebuild any feature by replaying the log), and natural fit for event-driven products. Weaknesses: the highest operational bar (state management, exactly-once, watermarks, backfills from a log are slow), a permanently running cost for every feature, and most models simply do not need it — a streaming-first plane for a churn model is engineering vanity.

Layer-by-layer comparison

The decision framework

Three variables decide it. Score each honestly and the architecture usually falls out.

Two rules break ties. First, start where the analytics platform already is — a W plane on an existing Snowflake is worth more this quarter than a perfect L plane next year, and the migration path W → L is well-trodden (export feature views to Iceberg; keep the SQL). Second, add S as a lane, not a replacement: nearly every mature plane is L with a streaming lane for the handful of features that need it. Pure S is rare and usually belongs to companies whose product is the stream.

GridDP's choice

Score GridDP: a platform team exists (the DPE guide built it), the data is large tabular plus a physical-telemetry firehose plus ticket text, reproducibility matters (fraud and pricing decisions are audited), and the tightest features — bid_depth_5m for pricing, device_velocity_24h for fraud at payment time — need seconds while everything else is fine at hourly. That is L with an S lane: Iceberg on S3 as the source of truth for feature history, Spark/dbt for batch features, Feast (Stack A) or the Databricks Feature Store (Stack B) as the registry and materializer, Redis as the online store, and one Flink job computing the six streaming features and writing them to both Redis and the lake. The warehouse engine (Trino or Databricks SQL) serves analysts off the same tables. MLflow pins dataset snapshots to model versions.

What GridDP explicitly does not do: put the firehose through the feature store (rollups happen in ClickHouse and land as hourly features), run pricing off batch (the S lane exists for exactly this), or build a second streaming path for churn (hourly is fine, and the always-on cost is not).

When to choose what

Finally, the whole plane assembled for GridDP, the mistakes to avoid, and where this is all heading — 14 · Reference Design for GridDP, Anti-patterns & Outlook.

✓ Check yourself

Answer from memory before opening each one.

1. The chapter says the three architectures answer one question. State it and give the answer for each, and say why it matters more than which feature store you choose.

   Show answer

   The question is what the source of truth is for a feature's history. For the warehouse-centric plane it is warehouse tables, for the lakehouse-centric plane it is open-format tables on object storage, and for the streaming-first plane it is the event log. Tooling, cost shape, latency floor and skills follow from that answer, and a vendor feature store does not decide it. See three ways to build the same plane.

2. A company of three ML engineers on Snowflake scores churn nightly. A consultant proposes a lakehouse with streaming. What do you say?

   Show answer

   Choose the warehouse-centric plane: the team is small and SQL-native, the model is tabular and an hour or a day of staleness is fine. Ship the first model this quarter, and migrate to a lakehouse later, since exporting feature views to Iceberg and keeping the SQL is a well-trodden path. See warehouse-centric.

3. Fraud needs signup velocity in seconds, and everything else tolerates an hour. Do you pick streaming-first? Say why or why not.

   Show answer

   No. Pick the lakehouse-centric plane with a streaming lane for the few features that need seconds. A fully streaming-first plane puts an always-on job behind every feature and carries the highest operating bar, which most models do not justify. See the decision framework.

4. Name three ways the warehouse is a poor fit for ML workloads, and what the plane does about each.

   Show answer

   It is a poor dataloader, since throughput to one client is modest, so write the training set to shards in the lake. It is a poor streaming engine, so add a stream path for sub-minute features. And per-query pricing punishes iterative experiments, so build datasets once. Unstructured data and vectors also sit poorly there. See warehouse-centric.

5. A team says they chose the Databricks feature store so the architecture is settled. What have they decided and what is still open?

   Show answer

   They chose tooling. They still need the online store, point-in-time discipline, skew checks and a streaming lane, because the vendor stores are thin over their table formats. The architectural choice is the source of truth for feature history, and the tool follows from it. See the decision framework.

6. Why does the chapter say starting where the analytics platform already is beats the ideal design, and what is the test for adding a streaming lane later?

   Show answer

   A working plane on the existing stack delivers models this quarter, while a perfect one next year delivers nothing now, and the migration paths between planes are routine. Add a streaming lane when a tier-1 feature's freshness need is seconds and the product's value depends on it. See the decision framework.

Takeaway

• The three planes differ in the source of truth for feature history: warehouse tables, open lakehouse tables or the event log.
• Warehouse-centric ships fastest for small SQL-native teams with tabular, hourly work, but is a poor dataloader, stream engine and home for vectors.
• Lakehouse-centric is the general shape for large, unstructured or audited work, with snapshot-pinned datasets and shards read straight by the trainer.
• Streaming-first gives structural parity and the lowest latency at the highest operating bar, and most models do not need it.
• Decide by team size, tightest latency need, budget shape, data shape and reproducibility bar, and start where the analytics platform already is.
• Add streaming as a lane for the few features that need it, and treat a vendor feature store as a tooling choice that follows the architecture.

With the fork chosen, the last chapter assembles the full plane for GridDP and names what to avoid. → Reference Design for GridDP, Anti-patterns & Outlook

Further reading

• Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics — Armbrust et al., CIDR 2021. The case for open formats on object storage as the system of record.
• Snowflake Feature Store — Snowflake documentation, accessed 2026. Feature views and entity definitions for the warehouse-centric plane.
• Databricks Feature Store — Databricks documentation, accessed 2026. Feature tables on Unity Catalog and point-in-time lookups for the lakehouse-centric plane.
• Feast architecture overview — Feast documentation, accessed 2026. A registry-only feature store that runs over a warehouse or a lake.
• Tecton documentation — Tecton, accessed 2026. A platform that owns stream and batch computation for the streaming lane.