Systems Design · Deep Dive

Data Platform Systems Design — GPU Marketplace

How a Data Platform Engineer designs the internal data platform for a GPU-marketplace company, end to end — from the application space and ingestion through lakehouse storage, data modeling, transformation, serving, governance, and AI enablement. Layered for both learners and senior architects. Built around GridDP, an illustrative platform for a hypothetical GPU marketplace, and three reference stacks compared throughout: open-source lakehouse, Databricks, and Snowflake.

How to read this

Start with 00 · Start Here — it defines the running case study (the company, its source systems, the canonical data model, the three reference stacks, and the scale numbers) that every later chapter builds on. Each chapter opens with foundational framing, then climbs to senior design-decision depth. If you already know the modern data stack, jump to 03 · Reference Architectures.

Section A · Orient & the problem space
00 Start Here The running case study — GridDP. The company, its source systems, the canonical data model, the three reference stacks, scale numbers, and reading order. Read first. 01 The Application Space Every source system in a GPU marketplace and the data it emits — marketplace txns, instance lifecycle, the GPU-telemetry firehose, bids/auctions, ranking, billing, KYC/crypto, support. Mapping the app space to data domains. 02 The Data Platform & the DPE Role What a data platform is; DPE vs DE vs analytics/platform engineering; platform-as-a-product; the stakeholder map across product, eng, finance, trust & safety, ML, and execs.
Section B · Architecture foundations
03 Reference Architectures Compared The three stacks side by side — open-source lakehouse vs Databricks vs Snowflake. Build-vs-buy, cost, team-size fit, and a decision framework. The backbone chapter. Backbone 04 Ingestion & Data Pipelines CDC from OLTP, event streaming, the telemetry firehose, batch vs streaming, ELT/ETL, schema registry, idempotency, late and out-of-order data. 05 Storage & the Lakehouse Parquet; Iceberg vs Delta vs Hudi; partitioning & clustering; object storage; hot/warm/cold tiers; the OLTP↔OLAP divide; a time-series store for telemetry. 06 Data Modeling Medallion layers; dimensional vs Data Vault vs One Big Table; slowly-changing dimensions; and the canonical GridDP data model / ERD — host, machine, GPU, instance, rental, bid, customer, invoice. 07 Transformation & Orchestration dbt; Airflow vs Dagster vs Prefect; incremental models; data tests; freshness SLAs; backfills and reprocessing.
Section C · Serving, quality & governance
08 Serving & the Semantic Layer BI, reverse ETL, and data APIs — plus the semantic / metrics layer as the single source of truth. The load-bearing layer for both self-serve and AI accuracy. Load-bearing 09 Self-Serve & Internal Enablement Supporting product & engineering — data catalog and discovery, data contracts, tiered datasets, access patterns, and onboarding internal users. 10 Data Quality, Observability & Reliability Tests, freshness/volume/schema monitors, lineage, data SLOs, and incident response for the platform itself. 11 Governance, Security & Compliance RBAC/ABAC, PII and KYC handling, multi-tenant customer-data isolation, data residency, audit, and cost governance.
Section D · AI enablement
12 The Platform for AI/ML Feature store, training-data pipelines, embeddings/vector storage, offline↔online parity, ML observability — powering ranking, fraud, churn, forecasting, and dynamic pricing. 13 AI Powering the Platform Text-to-SQL, semantic-layer-grounded NL analytics, RAG over the catalog, and agents for data tasks — the accuracy chapter: grounding, evals, guardrails, and why text-to-SQL fails without a semantic layer. Accuracy 14 Performance & Scale Engineering Query optimization, partitioning/clustering/Z-order, caching, materialization strategy, taming the telemetry firehose, and scaling the platform as GMV grows.
Section E · Putting it together
15 Reference Design: End-to-End The full worked platform for GridDP — architecture diagram, data flow, stack choices, org/team shape, a crawl→walk→run maturity roadmap, and capacity & cost math. 16 Synthesis, Anti-patterns & Outlook Best-practice recap, the top failure modes, the build/buy/maturity ladder, and where data platforms are heading — streaming-first and AI-native.
Reading paths
Full read (new to data platforms)

Read 00 → 16 in order. The case study compounds: the source systems in 01 become the ingestion problems in 04, the tables in 06, the marts in 08, and the features in 12. Don't skip 00 — every later chapter assumes its vocabulary.

Senior / architect (know the modern data stack)

Skim 00–02 for the case study, then go deep on 03 (architectures), 05 (lakehouse table formats), 06 (modeling), 08 (semantic layer), 11 (governance), 13 (AI accuracy), and 15 (the end-to-end design). These carry the load-bearing decisions.

"I need to enable AI on our data" path

00 (case study) → 06 (modeling) → 08 (semantic layer) → 12 (platform for ML) → 13 (AI powering the platform). The semantic layer in 08 is the hinge: it is what makes both self-serve analytics and AI-generated SQL trustworthy.