Self-Serve & Internal Enablement

The ticket-queue anti-pattern

Picture GridDP's first year. There are three data engineers and a dozen analysts. When Finance wants take-rate by region, they Slack the DPE team; someone writes the SQL, pastes a screenshot, and moves on. When Supply ops wants fill-rate by GPU model, same thing. It works — barely — because the data team can hold every table in its collective head.

Now apply the scale numbers from Start Here: tens to low hundreds of internal users across the six consumer groups in that chapter — Product & growth, Finance & execs, Supply ops, Trust & safety, ML & DS, and the AI/NL interfaces. Each of them has questions every day. If every question routes through the data team, the team is the platform — a human API with a queue depth that grows linearly with headcount and a latency measured in days.

Self-serve does not mean "throw everyone raw SQL access and wish them luck." Uncurated access produces the opposite failure: five analysts compute "active rentals" five different ways and walk into the same meeting with five GMV numbers. Real self-serve is a curated product — discoverable data (the catalog), trustworthy data (tiers and contracts), and consistent metrics (the semantic layer from chapter 08). The rest of this chapter builds those pieces.

Data catalog & discovery

The first wall a new analyst hits is not "I can't query the data" — it's "I can't find the data, and when I find a table I don't know whether to trust it." GridDP has thousands of tables across the lakehouse, the warehouse, and the ClickHouse telemetry store. Which one holds the canonical rental record? Is rentals_v2_final or fct_rental the one Finance uses? When was it last refreshed? Who owns it? Without answers, every analyst rediscovers the data model by trial and error — and half of them give up and Slack the data team, recreating the ticket queue.

A data catalog is the search-and-trust layer over the platform's metadata. Good discovery surfaces, for every dataset:

• Search & schemas — full-text search over table and column names, types, and descriptions, so "find me the bid table" actually works.
• Ownership — the team and on-call human responsible (tied to the domains from Start Here).
• Popularity & usage — how many people query it and how often; a proxy for "is this the real one."
• Freshness — last successful load and the expected cadence, so a stale table is visibly stale.
• Sample values & profiling — what the data actually looks like, null rates, distinct counts.
• Lineage — upstream sources and downstream consumers, so you can trace a number back to the source system and forward to every dashboard it feeds.
• Tier / certification — the trust badge (next section).

The catalog is itself a small data platform: harvesters crawl each store's system metadata (and the dbt manifest, and the BI tool's API) on a schedule or via push events, a metadata store (usually a graph, because lineage is a graph) holds the unified model, and a UI + API serve it to humans and — increasingly — to the NL/AI layer in chapter 13, which depends on rich catalog metadata to ground its answers.

Data contracts, operationalized

Chapter 01 introduced the data contract as the versioned agreement between a producing service and the platform — the API for what a source emits. Here we operationalize that idea and extend it one layer up: contracts also govern the datasets the platform publishes to its internal consumers. A contract is the seam that lets producer and consumer evolve independently, and the enforcement mechanism that keeps self-serve from turning into self-sabotage.

Four properties turn a contract from documentation into a load-bearing system:

• Producer-owned schema. The team that emits the data owns the contract, checked into their repo next to the code that produces it. The platform is a consumer with rights, not the author.
• CI enforcement. A schema change that violates the contract fails the producer's CI build — before it ships — not the platform's 3 a.m. pipeline. This is the whole point: the breaking change is caught where it's cheapest to fix.
• An explicit breaking-change policy. Additive changes are free; removals and type changes require a major-version bump and advance notice, so downstream consumers can migrate.
• The contract as the API boundary. Downstream code depends on the contract's guarantees (this field exists, is non-null, lands within 60s), not on the producer's internal table layout.

dataset: marketplace.fct_rental
owner: marketplace-domain          # ties to the domain in 00-START-HERE
tier: certified                     # raw | staged | certified (see below)
domain: marketplace
description: >
  One row per rental (a client's instance lifecycle on a host).
  The canonical rental grain. Use this, not rentals_v2_final.
sla:
  freshness: "<= 30 min behind source"
  availability: 99.5%
  on_call: "#marketplace-data-oncall"
schema:
  rental_id:    { type: string,    required: true, unique: true }   # grain key
  account_id:   { type: string,    required: true }                 # FK -> dim_account
  machine_id:   { type: string,    required: true }                 # FK -> dim_machine
  started_at:   { type: timestamp, required: true }                 # event_time, UTC
  ended_at:     { type: timestamp, required: false }                # null = still active
  gpu_hours:    { type: decimal,   required: true, min: 0 }
  revenue_usd:  { type: decimal,   required: true, min: 0 }         # reconciles to ledger
governance:
  pii: false
  certified_metrics: [gmv, take_rate, active_rentals]               # served via semantic layer (ch.08)
breaking_change_policy: major-version-bump + 2-week notice
tests:                                                              # enforced in CI (ch.10)
  - not_null: [rental_id, account_id, started_at]
  - unique: [rental_id]
  - relationships: { account_id: dim_account.account_id }

Dataset tiers & certification

Not every table deserves the same trust. The raw landing zone for the telemetry firehose and a Finance-reconciled GMV table are both "data," but depending on them carries wildly different risk. If the catalog treats them identically, analysts can't tell a battle-tested gold table from someone's abandoned scratch query. The answer is an explicit tier ladder — a visible trust signal that says how much weight a dataset can bear.

Promotion up the ladder is a deliberate act, not a default. A table earns the certified badge by acquiring an owner, an SLA, a contract, passing tests, and a human description — reviewed in a PR, the same way code earns a release tag. The badge then renders in the catalog as a trust signal: an analyst searching for "rental revenue" sees the certified fct_rental at the top and knows it's safe to put in front of the CFO.

How different personas access data

"Self-serve" means different things to a SQL analyst and a CFO. The platform has to meet each consumer group from Start Here on the interface they already live in — and crucially, route them all through the same certified datasets and the same semantic layer, so that five interfaces still produce one GMV number.

The design rule across the whole table: the interface varies, the source of truth does not. Reverse-ETL must read the certified metric, not re-derive it; the BI tool must hit the semantic layer, not hand-rolled SQL; the NL/AI bot must resolve "GMV" through the same metric definition the CFO's dashboard uses. The moment any interface computes its own version of a governed metric, the platform has forked its truth — exactly the failure self-serve was meant to prevent.

Onboarding internal users

A self-serve platform that takes a new analyst two weeks to figure out is not self-serve. The metric that captures this is time-to-first-query — how long from "new hire's first day" to "ran a useful query against a certified dataset, alone." Treating the platform as a product (the mindset from the DPE-role chapter) means optimizing that onboarding funnel as deliberately as the product team optimizes signup.

The enablement toolkit that drives time-to-first-query down:

• Golden paths — one blessed, documented way to do each common task: "how to build a dashboard," "how to add a dbt model," "how to query telemetry." Not ten options; one paved road that works.
• Starter templates — a dbt model scaffold, a notebook template pre-wired to the warehouse, a dashboard skeleton. New work starts from a known-good shape, not a blank page.
• Docs-as-code — documentation lives in the repo next to the code and ships with it, so it can't rot independently. Dataset descriptions in dbt YAML flow straight into the catalog.
• A "Datasets 101" guide — the canonical model, the tiers, the certified marts, and the three fact grains from Start Here, in one onboarding doc every new user reads.
• Office hours — a recurring, low-friction channel for the questions the docs don't cover. The trick is to treat each office-hours question as a signal: a recurring one means a missing golden path or dataset, and gets fixed at the source.

Federated ownership vs central platform

As the platform grows, one question gets sharper: who owns which datasets? The centralized model — one DPE team owns every pipeline and every table — is the ticket queue's cousin. It works early, then the central team becomes the bottleneck for every new dataset, because they're the only ones who can build one. The federated answer is data mesh: the domain teams own their data as products, and the central platform team provides the paved roads — the ingestion framework, the catalog, the semantic layer, the CI for contracts — on which domains build.

Mesh is a powerful organizing principle — but it has a real cost, and at GridDP's scale (tens to low hundreds of users, six domains, a data team of a handful), full federation can hurt more than it helps. Each domain owning a data product requires that domain to have someone who can build and operate one; spread that thin and you get six half-maintained pipelines instead of one good central one.

Measuring enablement

Enablement is a product, so you measure it like one. These metrics tell the DPE team whether the platform is actually getting more self-serve over time — or whether the ticket queue is quietly creeping back.

✓ Check yourself

Answer from memory before opening each one.

1. Recurring Slack requests for take-rate by region keep reaching the platform team. Leadership proposes hiring a second analyst to answer them faster. What do you propose instead, and how would you know it worked?

   Show answer

   Treat each recurring question as a platform bug: a missing certified dataset, metric or golden path. Fix the cause so people answer it themselves, and watch tickets per active user fall and the share of questions answered self-serve rise. A second human query API only moves the bottleneck. See the ticket-queue anti-pattern and measuring enablement.

2. Your catalog holds 4,000 tables, harvested automatically, and analysts still say they cannot find or trust anything. Which metadata came free, which did not, and what do you do about the second kind?

   Show answer

   Schemas, lineage and usage arrive free from harvesting. Descriptions, owners and tiers are human inputs, so an uncurated catalog is a phone book of strangers. Require owner, description and tier for certified datasets and track coverage, rather than asking everyone to document everything. See data catalog and discovery.

3. A producing team renames a column in a source table. Compare the cost of catching it in the producer's CI with catching it in the platform's overnight pipeline, and say what the contract must contain for CI to catch it.

   Show answer

   In the producer's CI the build fails before the change ships and the person who made it fixes it, while overnight it breaks pipelines and dashboards and someone else pages. CI needs a producer-owned schema in the producer's repo, a check against it, and a breaking-change policy that sends removals and type changes through a major version with notice. A contract not enforced in CI is a wish. See data contracts, operationalized.

4. A new table is landing useful data and an executive dashboard already depends on it. Should it be certified now? What must it acquire, and who reviews?

   Show answer

   Not yet. Certification is a reviewed promotion in a pull request that requires a named owner, a freshness and availability target, a CI-enforced contract, passing tests and a human description. Until then the dashboard is depending on an unguaranteed table, which is the risk the tiers make visible. See dataset tiers and certification.

5. Analysts use SQL, executives use dashboards, sales uses the CRM and everyone wants to ask questions in Slack. Do you build four metric definitions? Explain what varies and what does not.

   Show answer

   No. The interface varies by persona and the source of truth does not. Every interface reads the same certified core and semantic layer, and reverse ETL reads the certified metric rather than re-deriving it. See how different personas access data.

6. The ML team has deep data expertise and a backlog the central team cannot serve. A VP proposes mandating data mesh for all six domains. What do you argue?

   Show answer

   Federate ownership to the domains that are willing and able, such as ML owning feature datasets, and keep the central team on the paved roads and shared core. Mandating mesh at tens to low hundreds of users adds coordination cost and duplicated effort for domains without the staff to run it. See federated ownership versus central platform.

Takeaway

• The ticket queue — the DPE team as a human query API — does not scale past a handful of analysts; self-serve is the only architecture that grows sub-linearly with GridDP's tens-to-low-hundreds of users.
• Self-serve is a curated product: discoverable data (catalog), trustworthy data (tiers, certification, contracts), and consistent metrics (the semantic layer from ch. 08) — not raw SQL access for all.
• The catalog harvests metadata from every store into a graph and serves search, lineage, ownership, freshness, and trust badges to humans and the NL/AI layer alike.
• Data contracts, enforced in producer CI with an explicit breaking-change policy, are the API boundary that lets producers and consumers evolve independently.
• Tiers (raw → staged → certified) make trust visible; certification is a reviewed promotion, not a default. Meet each persona on its own interface, but ground them all on one certified core.
• Federate ownership to the willing and able domains; keep the central team owning the paved roads and shared core. Don't mandate mesh org-wide at this scale.
• Measure enablement like a product: time-to-first-query, % self-serve, catalog coverage, certified adoption, and the per-user ticket trend.

Self-serve only works if the data underneath is actually correct — and stays correct. Next we make trust enforceable rather than aspirational: tests, freshness, anomaly detection, and incident response. → Data Quality & Observability

Further reading

• How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh — Martin Fowler's site, 2019. The original argument for domain-owned data products and its costs.
• Data Contract Specification — datacontract.com. A concrete contract format with schema, quality and SLA sections.
• Model contracts — dbt documentation. Enforcing column names and types at build time.
• DataHub documentation — DataHub. Metadata ingestion, lineage and ownership in an open-source catalog.
• OpenMetadata documentation — OpenMetadata. The other open-source catalog named for Stack A.