Reference Design: End-to-End
This is the capstone. Every prior chapter built one organ; here we assemble the whole animal. We draw the full GridDP architecture as a single diagram, trace one real rental from a lifecycle event all the way to an exec dashboard and a churn feature, commit to the specific stack we'd actually choose and defend it layer by layer, sketch the team that runs it, lay out an 18-month roadmap, and do the capacity-and-cost math that decides where the money goes. One coherent system, from source to served metric.
The journey, in one breath
We started at the application space — nine sources, three shapes (OLTP, events, third-party) — and learned that the source dictates the pattern. We met the DPE role and the stakeholders, then weighed three reference architectures. We ingested via CDC, streaming, and batch; stored the lake on object storage with a specialist time-series engine beside it; modeled a two-sided dimensional spine with SCD2 history; transformed raw into bronze→silver→gold marts. We served trustworthy numbers through a semantic layer, enabled self-serve with contracts and a catalog, enforced quality and governance, built the platform for ML and one powered by AI, and made it all fast and affordable at scale. The thesis of the whole guide: a data platform is not a warehouse plus dashboards — it is a single pipeline of trust that turns physical events at 17,000 GPUs into a number a CFO will sign her name to and a feature a model will bet a churn prediction on. This chapter shows that pipeline as one machine.
The reference architecture
Here is the entire GridDP platform on one canvas. Read it left to right: sources are born on the left, flow through ingest into the dual-store core, get refined through the medallion transform tier, are named once in the semantic layer, and fan out to every consumer on the right. The single most important structural decision — the firehose taking the lower path into ClickHouse rather than through the warehouse — is visible as the bold lane.
The layers, each in a sentence (the chapter that designs it in parentheses):
- Sources (01). Nine systems in three shapes. The firehose is its own lane from the first inch.
- Ingest (04). Three patterns — Debezium CDC off Postgres WALs, Kafka streaming for events, batch/webhook pulls for third-party. The firehose streams but forks: a copy to the lake for history, the live path to ClickHouse for serving.
- Store (05). A dual store. An open lakehouse (Iceberg on S3) is the durable system of record for everything; ClickHouse is the purpose-built engine for the time-series firehose. The lake holds raw telemetry cheaply; ClickHouse holds queryable rollups.
- Transform (07). The medallion: bronze (raw, append-only landing), silver (deduped, conformed, SCD2 history merged from CDC), gold (the dimensional marts —
fct_rental_meter,dim_account,dim_machine, …). dbt owns this tier. - Semantic layer (08). The choke point of trust. Metrics like
gmv,take_rate_revenue, andfleet_utilizationare defined once over the gold marts so that BI, the SQL API, and the AI all compute the same number. - Serve (08, 12, 13). One semantic definition fans out to five surfaces: the BI tool, reverse-ETL back into product systems, a governed SQL/metrics API, the feature store (offline + online), and the NL/AI analytics interface.
- Govern & orchestrate (09–11). Catalog, contracts, tests, lineage, and access control are cross-cutting — they wrap every box. Dagster is the spine that runs and sequences it all.
If you remember one picture, remember the fork at ingest and the funnel at the semantic layer. The firehose forks early so the warehouse never drowns; everything funnels through the semantic layer so the company has one definition of every number. Those two decisions — divergence for cost, convergence for trust — are 80% of what makes this platform work.
One rental, end to end
Architecture diagrams hide the dynamics. So let's follow a single rental — call it rental_99c1, customer acct_7a renting machine m_8f21a's GPU index 3 — as its data threads through every layer twice: once down the analytics path to a GMV number on the CFO's dashboard, once down the ML path to a churn feature. Same source events, two destinations. This trace is the platform working.
Three things this trace makes concrete:
- The join is the product. Attributing physical telemetry (was the GPU actually delivering?) to a paying rental happens in silver, on
instance_idwithin an event-time window. Get the lifecycle timestamps wrong (ch. 01) and the meter, utilization, and churn feature all drift together. - One fact, two consumers.
fct_rental_meterfeeds both the GMV metric and the churn feature. They share a definition, which is why the dashboard and the model never disagree about how muchacct_7aspent. This is offline/online parity in practice. - Reconciliation closes the loop. The GMV number traces back through the semantic layer, the gold fact, silver, bronze, and CDC to a
ledger_entry— the penny-tie Finance demanded in ch. 01 and ch. 11. Lineage isn't documentation; it's the audit trail that lets the CFO trust the $1.42M.
When you onboard a new DPE, hand them this exercise: pick one rental and follow its bytes to a dashboard and a model. The moment they can narrate every hop — CDC, dedupe, SCD2 merge, the telemetry join, the meter, the metric definition, the feature — they understand the platform better than any architecture diagram could teach them. A platform you can't trace end-to-end is a platform you can't debug.
The stack we'd actually pick
The guide carried three reference stacks so we could reason about trade-offs without dogma. But this is the capstone, so we'll commit. For a company at GridDP's shape — a marketplace with a punishing telemetry firehose, a lean-but-real platform team, financial-grade correctness needs, and a strong allergy to vendor lock-in on its largest dataset — we'd build a pragmatic hybrid: an open lakehouse core (lean Stack A, Iceberg) for the durable system of record, ClickHouse for the firehose, a semantic layer over the top, and a feature store on the side. If the team were SQL-heavy and small with little platform headcount, we'd swap the core to Stack C (Snowflake) for time-to-value — the firehose still goes to ClickHouse either way. This is opinionated; reasonable teams choose differently.
| Layer | Our pick | Why this, here |
|---|---|---|
| Streaming bus | Kafka (or Redpanda) | The firehose plus all event sources need a durable, partitioned, replayable log. Redpanda if we want to cut ops overhead and JVM tuning. |
| CDC | Debezium | Log-based capture off the Postgres WAL is the only way to get full history without hammering the OLTP DBs. Open, battle-tested. |
| Table format | Apache Iceberg on S3 | The keystone choice. Our largest asset (telemetry history) stays in an open format on cheap object storage — no engine can hold it hostage. Multiple engines (Trino, Spark, ClickHouse) read the same tables. |
| Time-series engine | ClickHouse | Non-negotiable for the firehose. Serves "metric X for GPU Y over window W" 10–100× cheaper than a warehouse, with TTL'd rollups. The whole reason the warehouse survives. |
| Transform | dbt-core + Spark SQL | dbt is the lingua franca of the medallion; analysts can contribute. Spark SQL handles the heavy silver merges and rollups. |
| Query engine | Trino | Federated SQL over Iceberg and ClickHouse from one endpoint — analysts join business data to telemetry rollups without caring where each lives. |
| Orchestration | Dagster | Asset-aware orchestration models the lineage natively (CDC→silver→gold→metric), with freshness SLAs and backfills first-class. Airflow if the team already lives there. |
| Catalog / governance | Polaris/Nessie + DataHub | Polaris (or Nessie) is the Iceberg REST catalog; DataHub layers search, lineage, contracts, and PII tagging across both stores. |
| Semantic layer | dbt MetricFlow / Cube | One definition of every metric, queryable by BI, the API, and the LLM. Without it, NL-to-SQL hallucinates revenue. Cube if we need a caching API tier. |
| Feature store | Feast + MLflow | Offline features from the gold marts, online serving with parity, registry in MLflow. Reuses the lake — no separate feature pipeline. |
| BI | A semantic-aware tool (e.g. Lightdash/Superset on the metrics layer) | BI must query the semantic layer, never raw tables — that's how "one number" is enforced at the tool boundary. |
| NL / AI analytics | LLM grounded on the semantic layer + catalog | The LLM writes metric queries, not raw SQL; accuracy comes entirely from the curated metric and dimension definitions (ch. 13). |
The Iceberg-vs-Snowflake core is the real fork, and it's a capital and headcount decision, not a technical one (ch. 03). Open lakehouse: lower cloud bill, higher engineering payroll, zero lock-in on your biggest dataset. Snowflake core: faster to value, smaller team, a bill that grows with the firehose's rollups unless you're disciplined about keeping raw telemetry out. The one thing that does not diverge: the firehose goes to a specialist engine in every version. That's the load-bearing invariant.
Team & org shape
A platform is also an org design. The pattern that scales GridDP past tribal knowledge is a small central platform team that builds paved roads — the ingestion framework, the dbt project skeleton, the semantic layer, the catalog, the CI checks — and embedded analysts/data scientists who live in the business domains and ride those roads. The platform team owns the system; the domains own their data products and contracts.
Who owns what, RACI-style: the platform team is Accountable for ingestion, storage, the semantic layer's machinery, governance tooling, and uptime. Source-producing product teams are Accountable for their data contracts and schema stability (ch. 01, 09). Embedded analysts are Responsible for the gold marts and metric definitions in their domain; the platform team Consults on modeling and reviews PRs. ML teams are Responsible for features and models, with the platform team Accountable for the feature store's offline/online parity. Finance and execs are Informed consumers who sign off on metric definitions.
| Stage | Platform (DPE) | Embedded analysts | ML / DS | Total data org |
|---|---|---|---|---|
| Crawl (0–3 mo) | 2–3 | 1–2 | 0–1 | ~3–6 |
| Walk (3–9 mo) | 3–5 | 3–5 | 2–3 | ~8–13 |
| Run (9–18 mo) | 5–7 | 5–8 | 4–6 | ~15–20 |
A DPE's output is not dashboards — it's the paved road that lets twenty analysts and scientists ship without filing tickets. If the platform team becomes a request queue ("can you add this column?"), it has failed at org design no matter how good the tech is. Measure the platform team by how much self-serve work happens without them (ch. 09), not by their own ticket throughput.
The crawl → walk → run roadmap
You do not build the architecture diagram in one quarter. You build the spine first — CDC into a lake into a warehouse into a handful of marts and a BI tool — prove value, then layer on the semantic tier, governance, the firehose specialist, and finally the AI and optimization work. Sequencing by value unlocked, not by architectural elegance, is what keeps the project funded.
| Phase | What you build | Why now | What it unlocks |
|---|---|---|---|
| Crawl 0–3 mo |
Debezium CDC off the core Postgres DBs; Iceberg lake on S3; a query engine; first dbt project with bronze→silver→a few gold marts (fct_rental_meter, dim_account); one BI tool. |
The business needs some trustworthy number now. Start with the small, high-question relational domains — billing, marketplace — not the firehose. | First reconciled GMV/revenue dashboard. Proof the platform pays for itself. The spine everything else hangs on. |
| Walk 3–9 mo |
Semantic layer over the marts; data catalog + contracts; data-quality tests in CI; self-serve onboarding; feature store; route the firehose to ClickHouse with rollups into the lake. | Now that the spine works, the cost is duplicated metric logic, broken pipelines, and the firehose threatening the warehouse bill. Fix trust and tame the firehose. | One definition of every metric; analysts self-serve; first ML features with parity; fleet/utilization dashboards at firehose scale without warehouse blowup. |
| Run 9–18 mo |
NL/AI analytics grounded on the semantic layer; advanced governance (PII tokenization, audited access, lineage-based impact analysis); full observability (freshness, volume, schema drift); cost optimization (tiering, rollup tuning, autoscaling). | The platform is load-bearing for the company; now optimize for breadth of access, compliance, and unit economics. | Anyone asks questions in plain English; compliance-grade access; the platform is observable and cost-optimized at scale. |
Capacity & cost math
Design without magnitudes is fiction. Using the scale fixtures — ~17,000 GPUs, a sample every 10s, ~150 M telemetry samples/day, ~150–250 GB/day landed — here is roughly where the bytes and the dollars go. The numbers are order-of-magnitude teaching fixtures, but the shape is the lesson.
| Quantity | Math | Result | Design consequence |
|---|---|---|---|
| Telemetry rate | 17,000 × 6 / min | ~1,700 rows/s, ~150 M/day | The firehose. Sets everything below. |
| Raw landed growth | ~150–250 GB/day compressed | ~55–90 TB/yr | Object storage is cheap (~$23/TB-mo on S3); the lake absorbs this. Tier cold partitions to colder classes. |
| Telemetry rollup ratio | per-10s → per-GPU-hour | ~360× reduction | 150 M raw rows/day collapse to ~408 k rollup rows/day (17k × 24). This single step is what saves the warehouse. |
| Rollup in the warehouse | ~408 k rows/day × 365 | ~150 M rows/yr — trivial | The warehouse only ever sees rollups + business data — gigabytes, not petabytes. |
| ClickHouse footprint | raw at high compression + TTL | weeks of raw hot, then rolled + expired | TTL evicts raw detail after the ops window; the lake keeps the durable history. |
| Business data (CDC + events) | ~30–55 M changes/day | small vs. telemetry | ~80% of questions, ~1% of bytes. Cheap to model richly. |
Now the rough monthly cost shape (illustrative, mid-size cloud spend):
| Cost center | Driver | Rough share | Lever |
|---|---|---|---|
| Firehose ingest + ClickHouse compute | 1,700 rows/s, continuous | ~40–55% | Rollup aggressively; TTL raw; right-size CH nodes. |
| Lake storage (raw telemetry history) | ~55–90 TB/yr growing | ~10–15% | Tiering (hot→cold→archive); compression; partition pruning. |
| Warehouse / query compute (Trino + dbt) | marts + rollups + ad-hoc | ~20–30% | Keep firehose out; incremental models; autoscale. |
| Streaming bus (Kafka) | firehose + event throughput | ~10% | Retention tuning; tiered storage. |
| Catalog, BI, feature store, AI | seats + serving | ~5–10% | Mostly fixed; scales with users not bytes. |
Half the platform's bill traces to one source: the telemetry firehose. The reason the other half stays small is the ~360× rollup — collapsing per-10-second samples to per-GPU-hour before anything touches the warehouse. The signature failure mode (ch. 01, 05, 14) is skipping that step and piping raw telemetry into a columnar warehouse: the bill grows linearly with the fleet and the dashboards get slow. Spend your optimization energy where the money is — the firehose path — not on the cheap relational marts.
Takeaway
- The platform is one machine: sources → ingest → dual store → medallion transform → semantic layer → five serving surfaces, wrapped in governance and run by an orchestrator. Two decisions carry it — fork the firehose early for cost, funnel everything through the semantic layer for trust.
- You can trace a single rental from a lifecycle event and a telemetry sample, through bronze/silver/gold, to both a reconciled GMV number and a churn feature with offline/online parity. If you can narrate that trace, you understand the platform.
- The stack we'd actually build is a pragmatic hybrid: open Iceberg lakehouse core (or Snowflake if SQL-heavy and lean) + ClickHouse for the firehose + a semantic layer + a feature store. The firehose-to-specialist invariant never bends.
- Org and roadmap matter as much as tech: a small platform team building paved roads for embedded domain analysts, sequenced crawl → walk → run by value unlocked, funded by the math that says ~half the cost is the firehose and rollups are the lever.
One chapter remains. Having assembled the whole system, we step back to ask what's durable versus fashionable, where the field is heading, and what a DPE should bet on next. → Synthesis & Outlook