The Data Platform & the DPE Role
Chapter 01 mapped where data is born. This chapter defines the system that consumes that application space and the person who builds it: what a data platform actually is as a layered reference model, how a Data Platform Engineer differs from the half-dozen roles people confuse them with, why the platform should be run as an internal product, and which team topology fits GridDP at its current scale.
What a data platform actually IS
Ask five engineers what a "data platform" is and you'll get five tool lists — Snowflake, Kafka, dbt, Airflow, a BI tool. That's the inventory, not the definition. A data platform is the system that turns the application space into trustworthy, queryable, AI-ready data, and it's best understood not as a pile of tools but as a small number of capability layers that data flows through, wrapped by three cross-cutting concerns that touch every layer.
Four layers carry the data left-to-right; three concerns cut vertically through all of them:
One line each — the mental model to hold for the rest of the guide:
- Ingest — get data off the sources reliably and at the right latency (CDC, streaming, batch — ch. 04).
- Store — land it durably and cheaply in the shape each query pattern needs: object-store lake, time-series engine for the firehose, warehouse for business joins (ch. 05).
- Transform — clean, conform, model, and aggregate raw landings into trustworthy tables and metrics (ch. 06–07).
- Serve — expose the data to humans and machines through BI, a semantic layer, APIs, feature stores, and NL interfaces (ch. 08, 12, 13).
- Orchestrate — drive the whole pipeline on a schedule/DAG, handle retries and backfills (ch. 07).
- Govern — catalog, data contracts, access control, lineage, PII handling (ch. 09, 11).
- Observe — measure freshness, run quality tests, watch cost, and report against SLAs (ch. 10).
Every reference stack in Start Here — open-source lakehouse, Databricks, Snowflake — implements these same seven boxes; they just fill them with different products. Design at the layer level first and choose tools second. A DPE who reasons in layers can re-platform a vendor without re-architecting the company; one who reasons in tools is trapped the day pricing changes.
DPE vs the adjacent roles
"Data engineer" has fragmented into a half-dozen specialties, and the titles overlap enough that companies use them inconsistently. The distinctions are real, though — they're about which artifact you own and who your customer is. The clarifying question for any of these roles: do you build the road, or do you drive on it?
| Role | Primary artifact | Owns | Optimizes for | Typical tools |
|---|---|---|---|---|
| Data Platform Engineer (DPE) | The paved roads: ingestion framework, storage layout, catalog, CI, the self-serve substrate | The platform itself — reliability, cost, and the abstractions others build on | Leverage: enabling N teams without N× headcount | Iceberg/Delta, Kafka, Dagster, Terraform, Unity/Polaris, dbt framework |
| Data Engineer | Specific pipelines for a domain | Particular datasets land correctly and on time | Correctness & freshness of their tables | dbt, Spark/Flink, Airflow, SQL |
| Analytics Engineer | Modeled marts & metric definitions | Business logic in the warehouse — the dimensional & semantic layer | Trust & clarity of metrics for analysts | dbt, SQL, semantic layer, BI |
| ML Engineer | Models & serving / feature pipelines | Training data, features, online inference | Model quality & offline↔online parity | Feature store, MLflow, Python, Spark |
| Platform / Infra (DevOps / SRE) | Clusters, networks, CI/CD, IAM | The compute & cloud substrate under the data platform | Uptime, security, infra cost | Kubernetes, Terraform, cloud IAM, observability |
| Data Architect | Standards, target-state diagrams, decision records | Coherence of the design across teams | Long-term consistency & reduced lock-in | Modeling tools, ADRs, reference architectures |
Read the table top-down and the DPE's mandate sharpens: a Data Engineer ships a pipeline; the DPE ships the framework that makes every pipeline cheap to ship. An Analytics Engineer owns a mart; the DPE owns the warehouse, the contract tooling, and the test harness the mart depends on. The DPE is the role whose customers are other data people.
If your work removes the same toil from many teams at once — a one-line CDC connector instead of a bespoke pipeline, a templated dbt project instead of a blank repo, a catalog that answers "where does this number come from?" without a Slack thread — you're doing DPE work. If you're hand-building one domain's tables, you're doing data-engineering work using the platform. Most people do both; the title describes where the center of gravity sits.
At GridDP's scale — tens to low hundreds of internal users — these roles are not six separate people. One small team wears several hats. But knowing which hat you have on for a given task is what keeps the platform from collapsing into a pile of one-off pipelines, which is the single most common failure mode of an early data team.
Treating the platform as a product
The mindset that separates a platform that compounds from one that drowns is this: the internal data platform is a product, and its internal users are customers. That sounds like a slogan until you take it literally and notice everything it forces you to have:
- Users — named consumer groups (the six in the stakeholder map below), each with distinct jobs-to-be-done.
- SLAs — explicit promises: the revenue mart is fresh by 06:00, telemetry rollups lag ≤ 5 min, the catalog is the source of truth for ownership.
- A roadmap — capabilities sequenced by user value, not by whichever ticket shouted loudest.
- Adoption metrics — how many teams self-serve, query volume on certified marts, time-to-first-dashboard for a new analyst, % of pipelines on the paved road.
- Docs & onboarding — a new analyst is productive from a runbook, not from a DPE's calendar.
Contrast the two operating postures directly:
| Dimension | Ticket-driven ops | Platform-as-product |
|---|---|---|
| Unit of work | A request ("build me this table") | A capability ("self-serve modeling for a domain") |
| Scaling behavior | Linear — every user adds queue load | Sub-linear — capabilities serve many users |
| DPE's day | Reactive firefighting & bespoke pipelines | Building leverage; users unblock themselves |
| Quality signal | Ticket close rate | Adoption, SLA attainment, trust in metrics |
| Failure mode | Bus-factor of one; backlog grows forever | Over-engineering ahead of real demand |
A ticket queue scales linearly with users: every new analyst adds load to the same humans, and the DPE becomes a bottleneck the company routes around (shadow pipelines, exported CSVs, three conflicting revenue numbers). A product scales sub-linearly: a paved road built once serves the tenth analyst as cheaply as the first. With GridDP's tens-to-low-hundreds of users, you cannot ticket your way to coverage — self-serve is not a luxury, it's the only model that fits the headcount. This is why Start Here flagged "self-serve > ticket queue" against the user count.
The honest caveat: product thinking has its own failure mode — building a beautiful platform nobody asked for. The discipline is to let real consumer demand pull capabilities into existence, which is exactly why the stakeholder map comes next.
The stakeholder map
A platform exists for its consumers, and the six groups from Start Here want fundamentally different things from the same data. Designing for "the business" in the abstract produces a platform that serves no one well. Designing for these six jobs-to-be-done produces a platform with a roadmap.
| Stakeholder | Jobs-to-be-done | Key questions they ask | Platform capability needed |
|---|---|---|---|
| Product & growth | Move funnel, retention, marketplace liquidity | "Why did conversion drop? Which experiment won? Is supply meeting demand?" | Self-serve BI + trustworthy semantic/metrics layer (ch. 08–09) |
| Finance & execs | Report GMV, take-rate, margin, payouts | "What's revenue this quarter — the one number, tied to the ledger?" | Certified marts reconciled to the double-entry ledger (ch. 08, 11) |
| Supply ops | Keep the fleet healthy & full | "Which hosts are flaky? What's fill rate & utilization right now?" | Low-latency telemetry rollups + time-series serving (ch. 14) |
| Trust & safety / risk | Catch fraud, abuse, disputes | "Is this account fraudulent? Who touched this PII?" | Low-latency signals + audited, restricted access (ch. 11–12) |
| ML & data science | Ranking, fraud, churn, pricing, forecasting | "Give me training data + features with offline↔online parity." | Feature store, point-in-time-correct datasets (ch. 12) |
| AI agents & NL interfaces | Answer questions in Slack / natural language | "What was GMV last week, by region?" — asked by an LLM | A governed semantic layer the model can ground on (ch. 13) |
These groups pull in opposite directions on the same data. Supply ops wants telemetry fresh to the second; finance wants it correct to the penny and reconciled by morning. The AI agent's accuracy depends entirely on a semantic layer that finance and product must agree defines "revenue." A DPE's real job is mediating these tensions in the architecture — which is why one source (the firehose) ends up served by two stores with two different SLAs, a pattern we hit in ch. 05 and 14.
Operating model & team topology
How you staff the platform is itself a design decision, and it changes with company stage. Three topologies dominate, and the trade-off is always the same axis: central control & consistency vs. domain speed & context.
| Topology | Strength | Weakness | Best stage |
|---|---|---|---|
| Centralized | One standard, easy governance, no duplication | Central team becomes a request bottleneck; lacks domain context | Early / small (one team can hold it all) |
| Embedded | Fast, deep domain context, no queue | Silos, drift, duplicated effort, no shared platform | Mid-size with strong, independent product orgs |
| Hub-and-spoke (mesh) | Central paved roads + local domain ownership | Highest coordination cost; only pays off at scale | Large, many domains, mature platform |
At tens-to-low-hundreds of internal users, GridDP is squarely in centralized territory — a small platform team owns ingest, storage, the warehouse, and the catalog, and serves all six stakeholder groups directly. The senior move is to build that centralized team as if it will become a hub: paved roads, contracts (ch. 09), and self-serve from day one, so the eventual shift to hub-and-spoke is a staffing change, not a re-platform. Full-blown data mesh at this scale is premature — it adds coordination overhead the org can't yet amortize.
A preview: crawl / walk / run
You do not build all seven layers at full fidelity on day one. The platform matures along a crawl → walk → run path, and knowing which phase you're in keeps you from over-building (the product-thinking failure mode) or under-building (the ticket-queue one). Chapter 15 develops the full capability maturity model; here is the shape.
| Capability | Crawl | Walk | Run |
|---|---|---|---|
| Ingest | Batch dumps, some manual | CDC + streaming on a framework | Self-serve connectors, contract-enforced |
| Store / model | Raw tables, ad-hoc SQL | Dimensional model, layered lake | Tiered storage, firehose on a TS engine |
| Serve | A few dashboards | Semantic layer + self-serve BI | Feature store + governed NL/AI access |
| Govern / observe | Tribal knowledge | Catalog, tests, basic lineage | Contracts in CI, SLAs, cost observability |
| Operating model | One person, tickets | Centralized platform team | Hub-and-spoke, self-serve domains |
Where is GridDP on this map?
The guide designs GridDP at a solid walk, reaching toward run on the dimensions that the marketplace's economics force: the telemetry firehose pushes store and serve toward "run" early (a time-series engine and rollups are not optional at 17,000 GPUs), while the operating model stays at "walk" (a centralized team) because the user count doesn't yet justify spokes. Maturity is rarely uniform across layers — you advance the capabilities your scale numbers demand first.
Takeaway
- A data platform is a stable set of capability layers — ingest → store → transform → serve, wrapped by orchestrate / govern / observe — not a tool list. Design at the layer level; choose tools second.
- The DPE builds the paved roads others drive on: the framework, not the individual pipeline. Their customers are the other data roles — DE, AE, MLE, analysts, and AI agents.
- Run the platform as an internal product — users, SLAs, a roadmap, adoption metrics, docs — because a ticket queue scales linearly with users and self-serve does not.
- Serve the six stakeholder groups by their jobs-to-be-done; their conflicting SLAs are the architecture's hardest constraint.
- GridDP's scale calls for a centralized team built like a future hub, sitting at "walk" overall but pushed toward "run" on storage and serving by the firehose.
With the application space mapped (ch. 01) and the platform and role defined, we can finally design the system. Next: the reference architectures — why you'd choose the open-source lakehouse, Databricks, or Snowflake stack, and the trade-offs at every layer.