Reference Architectures Compared

The seven capabilities every architecture has

Before naming a single product, internalize this: a data platform is not a pile of tools, it is a set of capabilities that must be filled somehow. Every architecture — Snowflake, Databricks, a hand-rolled open-source lakehouse, or the platform inside Google — fills the same seven slots. The vendors differ only in how many slots one product covers and how tightly those slots are welded together.

Read left to right, the first three are the data path: ingest moves bytes in, store holds them durably, transform turns raw landings into trustworthy models. The last four are cross-cutting: orchestration drives the path, serving exposes the output, governance keeps it legal and trustworthy, and observability tells you when it breaks. Later chapters take one slot each — ingest (04), store (05), modeling and transform (06–07), serving (08), governance (09–11), observability (10).

Stack A — Open-source lakehouse

The "own everything" stack. You assemble best-of-breed open components on object storage, run them yourself (on Kubernetes or managed equivalents), and pay almost nothing in license fees — trading that cash for engineering headcount. This is the stack large, infra-capable teams reach for, and it is the most natural home for GridDP's telemetry firehose because the time-series specialist (ClickHouse) is a first-class citizen rather than a bolt-on.

How the slots are filled: ingest is Debezium for CDC and Flink for stream processing, all flowing through Kafka or Redpanda as the bus. Store is Apache Iceberg tables on S3 (the open table format that gives you ACID, time-travel, and schema evolution without a proprietary engine), with the firehose forked off to ClickHouse. Transform is dbt-core compiling SQL that runs on Spark or Trino. Orchestrate is Dagster. Govern is an Iceberg catalog (Polaris or Nessie) plus DataHub for lineage and PII tagging. ML is Feast plus MLflow.

Stack B — Databricks-centric

The "lakehouse, managed, ML-first" stack. Databricks gives you a single platform built on open Delta Lake, with Spark as the compute substrate for everything — batch, streaming, and ML. You still write code (notebooks, SQL, Python), but the cluster management, table format, catalog, and ML tooling are integrated and operated for you. This is the stack to pick when machine learning and large-scale Spark transformation are central to the product — which, for a marketplace computing DLPerf scores, reliability models, and dynamic pricing, they are.

Slot fill: ingest is Auto Loader and Delta Live Tables on Structured Streaming (with Debezium upstream for CDC). Store is Delta Lake — open-format, ACID, with the bonus that it is the same format the ML training reads. Transform is DLT or dbt, both on Spark. Orchestrate is Databricks Workflows. Govern is Unity Catalog — the genuine differentiator: one catalog governing tables, files, ML models, and notebooks with unified lineage and access control. Serve is Databricks SQL with the Photon engine; NL-to-SQL is Genie. ML is the tightest of the three stacks: Feature Store, MLflow, and Mosaic all live where the data already is.

Stack C — Snowflake-centric

The "managed warehouse, analyst-first" stack. Snowflake separates storage from compute, manages essentially everything, and gives analysts the cleanest SQL experience of the three. You provision virtual warehouses (compute that spins up in seconds, scales independently, and auto-suspends), point connectors at your sources, and write SQL. It is the fastest path from zero to a working analytics platform — which is why it wins on time-to-value and is the default for analyst-heavy organizations.

Slot fill: ingest is Snowpipe Streaming for events and Fivetran-style managed connectors for OLTP and third-party APIs. Store is Snowflake's native tables (the FDN format) plus Iceberg for openness where you want it. Transform is dbt Cloud plus Dynamic Tables (declarative, incrementally-refreshed materializations — you declare the target SQL and a freshness target, Snowflake maintains it). Orchestrate is Tasks. Govern is Horizon. Serve is virtual warehouses; NL-to-SQL is Cortex Analyst, the most polished of the three. ML is Snowpark plus the Feature Store.

The big side-by-side

Here is the head-to-head across the dimensions that actually decide the choice. Ratings are relative (★ = weak fit, ★★★★★ = best-in-class) for GridDP's profile — a mid-stage two-sided marketplace with a heavy telemetry firehose and a small-to-medium analyst/ML team.

A decision framework

Reduce the comparison to a small number of questions answered in order. Each prunes the space; the first one that gives a confident answer usually settles it. Walk the tree top-down for GridDP and you can predict where it lands.

For GridDP specifically: lock-in is a concern but not an absolute mandate (Q1 → no); the company can fund a small platform team but it is not large (Q2 → marginal); ML is central — DLPerf, reliability, fraud, and dynamic pricing are product features, not back-office reports (Q3 → yes). That routes the business-data platform toward Stack B, with strong arguments for C on the analyst side — and, as the final ALWAYS rule states, the firehose is split out no matter what. In other words, the framework predicts a hybrid, which is exactly what real companies build.

Why real companies run hybrids, not one pure stack

The reference stacks are pedagogical extremes. No mature company runs a religiously pure A, B, or C. They run a dominant main platform plus a small number of deliberate specialists — and at a telemetry-heavy company the most important specialist is forced on you by physics, not preference.

This is the same 80/20 asymmetry from Start Here, now expressed as architecture. The relational business data — accounts, rentals, bids, invoices, payments — is small, high-value, heavily joined, and the source of most questions; it belongs in the warehouse/lakehouse. The telemetry firehose is huge, time-series, and read with range-scan queries ("util for GPU X over window W"); it belongs in a column-oriented time-series engine. Only pre-aggregated rollups (per-GPU-hour) cross from the firehose into the main platform, where they join to billing and reliability. We design that boundary in detail in chapters 05 and 14.

Build vs buy

Every slot you fill is a build-or-buy decision. Getting these right is most of what separates a platform that scales from one that drowns in maintenance. The rule of thumb: buy the commoditized, undifferentiated heavy lifting; build only the thin layer that encodes your company's specific knowledge.

A concrete "worth building" example: a golden-path scaffold so any engineer creates a contract-checked, lineage-registered dataset with one command, instead of hand-wiring dbt, the catalog, and tests each time.

# `griddp new-dataset` scaffolds the thin glue, not the engines
dataset: fleet.gpu_hourly_utilization
domain: supply            # → maps to catalog namespace + ACL group (ch. 11)
source: clickhouse.telemetry_sample
materialize:
  engine: dbt             # we BUY dbt; we BUILD the convention around it
  grain: gpu_id, hour
  freshness_sla: 30m      # → wired into the observability checks (ch. 10)
contract:
  owner: supply-ops
  pii: false
  tests: [not_null(gpu_id), accepted_range(util_pct, 0, 100)]
emits_rollup_to: warehouse.gold.fct_telemetry_hourly   # the firehose↔warehouse boundary

Takeaway

• Every data architecture fills the same seven capability slots — ingest, store, transform, orchestrate, serve, govern, observe. You are choosing how to fill them, not which tool is universally "best."
• Stack A (OSS lakehouse) minimizes license cost and lock-in and is the best home for the firehose, at the price of the highest engineering headcount.
• Stack B (Databricks) wins when ML and large-scale Spark are central, with the strongest unified governance — costlier and engineer-shaped.
• Stack C (Snowflake) wins on time-to-value, analyst ergonomics, and NL-to-SQL — but the telemetry firehose is awkward and expensive on it.
• No stack wins every dimension; the choice is a weighted sum, and a short decision tree (lock-in mandate? platform team? ML-central? analyst-first? capital model?) settles it.
• Real companies — and GridDP — run hybrids: a main warehouse/lakehouse for business data plus a ClickHouse-style time-series engine carved out for the ~150 M-samples/day firehose. The firehose forces this split regardless of the main stack.
• Never build storage engines or CDC; buy the commodity slots; build only the thin glue (semantic layer, golden paths, the firehose↔warehouse boundary) that encodes your domains. And remember the TCO illusion: "free" open source is paid for in salaries.

With the architecture chosen at the slot level, we descend into the first slot in earnest. → Ingestion & Pipelines — CDC, streaming, and batch, applied to all nine sources.