Start Here

What this guide is (and isn't)

This is a guide about designing the data platform itself — the ingestion, storage, modeling, transformation, serving, governance, and AI-enablement systems that a Data Platform Engineer (DPE) stands up so that everyone else in the company can answer questions with data and ship ML. It is a systems-design guide: it reasons about components, data flows, trade-offs, and failure modes.

It is not a guide about doing data-engineering tasks on rented GPUs, and it is not an interview cram sheet. For the former, read the sibling guide Data Engineering for Neoclouds.

The case study: GridDP

Throughout the guide we design GridDP — the internal data platform for a hypothetical GPU-rental marketplace. A quick refresher on why a marketplace is a fascinating data-platform case study:

• It is two-sided. Providers (GPU owners) list machines; clients rent them. Every event has two economic actors, which doubles the modeling and the metrics.
• The product emits a firehose of physical telemetry. Tens of thousands of GPUs report utilization, temperature, power, and benchmark scores every few seconds. This is the high-volume, time-series part of the platform.
• Pricing is a live market. On-demand prices and interruptible bids change by the hour. The platform needs event-time-correct pricing and clearing data.
• Trust is system-generated. DLPerf benchmarks, reliability scores, and fraud signals are data products the platform computes and feeds back into the marketplace ranking — the data platform is in the product's critical path, not just a back office.
• Money flows both ways. Clients are billed; providers are paid out; crypto and fiat rails, KYC, and a double-entry ledger all generate financial data with correctness requirements.

The source systems (the application space)

A data platform is only as good as its understanding of where data is born. Chapter 01 goes deep on this; here is the map you should hold in your head for the whole guide. GridDP ingests from nine source systems:

Two properties of this source map drive most of the architecture:

1. Heterogeneous shapes. Slowly-changing relational tables (accounts), high-frequency append-only events (telemetry), and external API pulls (payments) need fundamentally different ingestion patterns — CDC, streaming, and batch respectively. Chapter 04 is organized around exactly this split.
2. One source dominates volume. The host-agent telemetry firehose is 2–3 orders of magnitude larger than everything else combined. A huge fraction of the platform's cost and design tension comes from this single source. Watch it recur in 04 (ingest), 05 (store), 14 (scale).

The data domains

Source systems are an engineering view. The platform also needs a business view — the domains that analysts and ML models actually reason about. We organize GridDP into six domains; most chapters touch one or two explicitly.

Domains are not just documentation — they become the ownership boundaries for data contracts (chapter 09), the seams for access control (chapter 11), and the top-level namespaces in the catalog. A platform without explicit domains becomes a swamp where no one owns anything.

The canonical data model

Here is the core entity model in miniature. Chapter 06 develops the full dimensional model (with slowly-changing dimensions, fact grains, and DDL); this is the skeleton to memorize first. Note the two-sided spine: supply on the left, demand on the right, meeting at the rental.

The three reference stacks

Rather than prescribe one toolchain, the guide carries three end-to-end reference architectures and compares them at every layer. When a chapter says "in Stack A…", this is what it means:

Chapter 03 explains why you would choose each (capital model, team size, lock-in tolerance, cost curve). The short version: A minimizes vendor cost and lock-in at the price of engineering headcount; B excels when ML and large-scale Spark are central; C wins on time-to-value and SQL-analyst ergonomics. A real company at GridDP's stage usually runs a hybrid — and a marketplace's telemetry firehose pushes all three toward a specialist time-series engine (ClickHouse) on the side.

Scale numbers (the fixtures)

Design is meaningless without magnitudes. These are the numbers we size GridDP against; they recur in the capacity math in chapters 04, 05, 14, and 15.

Who consumes the data

A platform exists for its consumers. GridDP serves five, and chapters 08–13 are largely about serving them well:

• Product & growth — funnels, retention, marketplace liquidity, pricing experiments. Want self-serve dashboards and a trustworthy metrics layer (ch. 08–09).
• Finance & execs — GMV, take-rate revenue, margins, payouts. Want one number for each metric, reconciled to the ledger (ch. 08, 11).
• Supply ops — fleet health, utilization, fill rate, provider economics. Heavy telemetry consumers (ch. 14).
• Trust & safety / risk — fraud, abuse, disputes. Want low-latency signals and audited access to sensitive data (ch. 11–12).
• ML & data science — ranking, fraud, churn, capacity forecasting, dynamic pricing. Want features, training data, and offline↔online parity (ch. 12).
• AI agents & NL interfaces — increasingly, the "user" is an LLM answering questions in Slack. Accuracy depends entirely on the semantic layer (ch. 13).

Reading order

The chapters are grouped into five sections that follow the data's journey:

• Section A · Orient (01–02) — the application space and the DPE role.
• Section B · Architecture foundations (03–07) — architectures, ingestion, storage, modeling, transformation. The load-bearing build chapters.
• Section C · Serving, quality & governance (08–11) — getting trustworthy data to internal users.
• Section D · AI enablement (12–14) — the platform for ML, AI powering the platform, and performance at scale.
• Section E · Putting it together (15–16) — the end-to-end reference design and synthesis.

Read in order if you're new to the space. If you're a senior engineer, the recommended deep-dive path is in the guide hub. Next up: the application space — every source system, in detail.