Self-Serve & Internal Enablement

The ticket-queue anti-pattern

Picture GridDP's first year. There are three data engineers and a dozen analysts. When Finance wants take-rate by region, they Slack the DPE team; someone writes the SQL, pastes a screenshot, and moves on. When Supply ops wants fill-rate by GPU model, same thing. It works — barely — because the data team can hold every table in its collective head.

Now apply the scale numbers from Start Here: tens to low hundreds of internal users across the six consumer groups in that chapter — Product & growth, Finance & execs, Supply ops, Trust & safety, ML & DS, and the AI/NL interfaces. Each of them has questions every day. If every question routes through the data team, the team is the platform — a human API with a queue depth that grows linearly with headcount and a latency measured in days.

Self-serve does not mean "throw everyone raw SQL access and wish them luck." Uncurated access produces the opposite failure: five analysts compute "active rentals" five different ways and walk into the same meeting with five GMV numbers. Real self-serve is a curated product — discoverable data (the catalog), trustworthy data (tiers and contracts), and consistent metrics (the semantic layer from chapter 08). The rest of this chapter builds those pieces.

Data catalog & discovery

The first wall a new analyst hits is not "I can't query the data" — it's "I can't find the data, and when I find a table I don't know whether to trust it." GridDP has thousands of tables across the lakehouse, the warehouse, and the ClickHouse telemetry store. Which one holds the canonical rental record? Is rentals_v2_final or fct_rental the one Finance uses? When was it last refreshed? Who owns it? Without answers, every analyst rediscovers the data model by trial and error — and half of them give up and Slack the data team, recreating the ticket queue.

A data catalog is the search-and-trust layer over the platform's metadata. Good discovery surfaces, for every dataset:

• Search & schemas — full-text search over table and column names, types, and descriptions, so "find me the bid table" actually works.
• Ownership — the team and on-call human responsible (tied to the domains from Start Here).
• Popularity & usage — how many people query it and how often; a proxy for "is this the real one."
• Freshness — last successful load and the expected cadence, so a stale table is visibly stale.
• Sample values & profiling — what the data actually looks like, null rates, distinct counts.
• Lineage — upstream sources and downstream consumers, so you can trace a number back to the source system and forward to every dashboard it feeds.
• Tier / certification — the trust badge (next section).

The catalog is itself a small data platform: harvesters crawl each store's system metadata (and the dbt manifest, and the BI tool's API) on a schedule or via push events, a metadata store (usually a graph, because lineage is a graph) holds the unified model, and a UI + API serve it to humans and — increasingly — to the NL/AI layer in chapter 13, which depends on rich catalog metadata to ground its answers.

Data contracts, operationalized

Chapter 01 introduced the data contract as the versioned agreement between a producing service and the platform — the API for what a source emits. Here we operationalize that idea and extend it one layer up: contracts also govern the datasets the platform publishes to its internal consumers. A contract is the seam that lets producer and consumer evolve independently, and the enforcement mechanism that keeps self-serve from turning into self-sabotage.

Four properties turn a contract from documentation into a load-bearing system:

• Producer-owned schema. The team that emits the data owns the contract, checked into their repo next to the code that produces it. The platform is a consumer with rights, not the author.
• CI enforcement. A schema change that violates the contract fails the producer's CI build — before it ships — not the platform's 3 a.m. pipeline. This is the whole point: the breaking change is caught where it's cheapest to fix.
• An explicit breaking-change policy. Additive changes are free; removals and type changes require a major-version bump and advance notice, so downstream consumers can migrate.
• The contract as the API boundary. Downstream code depends on the contract's guarantees (this field exists, is non-null, lands within 60s), not on the producer's internal table layout.

dataset: marketplace.fct_rental
owner: marketplace-domain          # ties to the domain in 00-START-HERE
tier: certified                     # raw | staged | certified (see below)
domain: marketplace
description: >
  One row per rental (a client's instance lifecycle on a host).
  The canonical rental grain. Use this, not rentals_v2_final.
sla:
  freshness: "<= 30 min behind source"
  availability: 99.5%
  on_call: "#marketplace-data-oncall"
schema:
  rental_id:    { type: string,    required: true, unique: true }   # grain key
  account_id:   { type: string,    required: true }                 # FK -> dim_account
  machine_id:   { type: string,    required: true }                 # FK -> dim_machine
  started_at:   { type: timestamp, required: true }                 # event_time, UTC
  ended_at:     { type: timestamp, required: false }                # null = still active
  gpu_hours:    { type: decimal,   required: true, min: 0 }
  revenue_usd:  { type: decimal,   required: true, min: 0 }         # reconciles to ledger
governance:
  pii: false
  certified_metrics: [gmv, take_rate, active_rentals]               # served via semantic layer (ch.08)
breaking_change_policy: major-version-bump + 2-week notice
tests:                                                              # enforced in CI (ch.10)
  - not_null: [rental_id, account_id, started_at]
  - unique: [rental_id]
  - relationships: { account_id: dim_account.account_id }

Dataset tiers & certification

Not every table deserves the same trust. The raw landing zone for the telemetry firehose and a Finance-reconciled GMV table are both "data," but depending on them carries wildly different risk. If the catalog treats them identically, analysts can't tell a battle-tested gold table from someone's abandoned scratch query. The answer is an explicit tier ladder — a visible trust signal that says how much weight a dataset can bear.

Promotion up the ladder is a deliberate act, not a default. A table earns the certified badge by acquiring an owner, an SLA, a contract, passing tests, and a human description — reviewed in a PR, the same way code earns a release tag. The badge then renders in the catalog as a trust signal: an analyst searching for "rental revenue" sees the certified fct_rental at the top and knows it's safe to put in front of the CFO.

How different personas access data

"Self-serve" means different things to a SQL analyst and a CFO. The platform has to meet each consumer group from Start Here on the interface they already live in — and crucially, route them all through the same certified datasets and the same semantic layer, so that five interfaces still produce one GMV number.

The design rule across the whole table: the interface varies, the source of truth does not. Reverse-ETL must read the certified metric, not re-derive it; the BI tool must hit the semantic layer, not hand-rolled SQL; the NL/AI bot must resolve "GMV" through the same metric definition the CFO's dashboard uses. The moment any interface computes its own version of a governed metric, the platform has forked its truth — exactly the failure self-serve was meant to prevent.

Onboarding internal users

A self-serve platform that takes a new analyst two weeks to figure out is not self-serve. The metric that captures this is time-to-first-query — how long from "new hire's first day" to "ran a useful query against a certified dataset, alone." Treating the platform as a product (the mindset from the DPE-role chapter) means optimizing that onboarding funnel as deliberately as the product team optimizes signup.

The enablement toolkit that drives time-to-first-query down:

• Golden paths — one blessed, documented way to do each common task: "how to build a dashboard," "how to add a dbt model," "how to query telemetry." Not ten options; one paved road that works.
• Starter templates — a dbt model scaffold, a notebook template pre-wired to the warehouse, a dashboard skeleton. New work starts from a known-good shape, not a blank page.
• Docs-as-code — documentation lives in the repo next to the code and ships with it, so it can't rot independently. Dataset descriptions in dbt YAML flow straight into the catalog.
• A "Datasets 101" guide — the canonical model, the tiers, the certified marts, and the three fact grains from Start Here, in one onboarding doc every new user reads.
• Office hours — a recurring, low-friction channel for the questions the docs don't cover. The trick is to treat each office-hours question as a signal: a recurring one means a missing golden path or dataset, and gets fixed at the source.

Federated ownership vs central platform

As the platform grows, one question gets sharper: who owns which datasets? The centralized model — one DPE team owns every pipeline and every table — is the ticket queue's cousin. It works early, then the central team becomes the bottleneck for every new dataset, because they're the only ones who can build one. The federated answer is data mesh: the domain teams own their data as products, and the central platform team provides the paved roads — the ingestion framework, the catalog, the semantic layer, the CI for contracts — on which domains build.

Mesh is a powerful organizing principle — but it has a real cost, and at GridDP's scale (tens to low hundreds of users, six domains, a data team of a handful), full federation can hurt more than it helps. Each domain owning a data product requires that domain to have someone who can build and operate one; spread that thin and you get six half-maintained pipelines instead of one good central one.

Measuring enablement

Enablement is a product, so you measure it like one. These metrics tell the DPE team whether the platform is actually getting more self-serve over time — or whether the ticket queue is quietly creeping back.

Takeaway

• The ticket queue — the DPE team as a human query API — does not scale past a handful of analysts; self-serve is the only architecture that grows sub-linearly with GridDP's tens-to-low-hundreds of users.
• Self-serve is a curated product: discoverable data (catalog), trustworthy data (tiers, certification, contracts), and consistent metrics (the semantic layer from ch. 08) — not raw SQL access for all.
• The catalog harvests metadata from every store into a graph and serves search, lineage, ownership, freshness, and trust badges to humans and the NL/AI layer alike.
• Data contracts, enforced in producer CI with an explicit breaking-change policy, are the API boundary that lets producers and consumers evolve independently.
• Tiers (raw → staged → certified) make trust visible; certification is a reviewed promotion, not a default. Meet each persona on its own interface, but ground them all on one certified core.
• Federate ownership to the willing and able domains; keep the central team owning the paved roads and shared core. Don't mandate mesh org-wide at this scale.
• Measure enablement like a product: time-to-first-query, % self-serve, catalog coverage, certified adoption, and the per-user ticket trend.

Self-serve only works if the data underneath is actually correct — and stays correct. Next we make trust enforceable rather than aspirational: tests, freshness, anomaly detection, and incident response. → Data Quality & Observability