AI Powering the Platform

AI: the newest data consumer — the promise and the peril

Every chapter so far has served a consumer with a known shape: a finance analyst wants a governed dashboard, the product wants a low-latency score, an ML model wants features. Now a new consumer arrives, and it does not fit the mold. Someone in the supply-ops channel types "what was H100 fill rate last week?" and expects a number back in seconds — no ticket, no analyst, no dashboard. This is the dream the Start Here consumer list flagged: the "user" is increasingly an LLM, and the value is enormous. Self-serve finally means everyone, not just people who write SQL.

That is the promise. The peril is sharper than for any human consumer. A dashboard that fails renders an error; a human analyst who is unsure says "let me check." An LLM does neither by default — it returns a fluent, plausible, confidently wrong number, with no signal that anything went wrong. And a wrong number in Slack is worse than no number, because it gets screenshotted, quoted in a standup, and pasted into a board doc before anyone reconciles it. The executive-trust crisis from ch. 08 — "but is this one right?" — does not go away with AI; it accelerates.

The naive answer — "just point an LLM at the warehouse and let it write SQL" — fails for exactly the reason ch. 08 spent a whole chapter on. The warehouse is full of raw tables with ambiguous columns, undocumented joins, and five plausible ways to compute every metric. A model handed that surface area does not become a careful analyst; it becomes the sixth team in the five-way "active GPU" fight, inventing a definition on the fly. Accuracy is not a model property. It is a platform property, and we build it the same way we built trust for humans: by removing the model's ability to be wrong, not by hoping it gets smarter.

Text-to-SQL: how it works, and why raw schemas break it

The core mechanism is text-to-SQL: give the model a natural-language question plus a description of the database, and have it emit a SQL query. In its naive form the "description" is the raw schema — table names, column names, maybe a few foreign keys — pulled from the warehouse catalog.

It demos beautifully on a clean three-table toy schema and collapses on a real platform. The reason is that every hard part of analytics lives in knowledge that is not in the schema. The column util does not say whether it means allocated time or actual compute. The path from rental to gpu does not say which of three join keys is correct. Nothing in the DDL encodes that "fill rate" means rented GPU-hours over listed GPU-hours and not over total hours. The model fills every gap with a confident guess, and crucially, it has no notion of correctness — it optimizes for plausible SQL, not for the right answer. Recall ch. 08's "five definitions of active GPU": a raw-schema model will reproduce that chaos query by query, never converging on the one the company agreed to.

The unifying point: these are not bugs a better model fixes. They are missing-context failures. The knowledge to answer correctly exists in the company — it just isn't in the schema the model was handed. The rest of this chapter is about handing the model the right context, and the single highest-leverage piece of context is the semantic layer.

Grounding on the semantic layer = the accuracy lever

Here is the central move. Instead of asking the model to write arbitrary SQL against raw tables, we ask it to pick a metric and some dimensions from the governed semantic layer — and let the layer compile the correct SQL, exactly as it does for every other consumer in ch. 08. The model never touches the join, the filter, or the aggregation. It chooses from a validated menu, and the menu guarantees the answer.

This is the whole trick, and it is a problem-space collapse. "Write any SQL over 200 tables" is an essentially unbounded space in which almost every point is subtly wrong. "Pick a metric from a list of nine and some dimensions from a known set" is a tiny, enumerable space in which every point is correct by construction, because the semantic layer authored the SQL, not the model. We have not made the model smarter. We have changed its job from authoring correctness (which it cannot do) to selecting from pre-validated correctness (which it does well).

// "what was H100 fill rate last week?"
// The LLM does NOT emit SQL. It emits a semantic-layer query:
{
  "metric": "fill_rate",          // chosen from the ch.08 metric catalog
  "dimensions": ["gpu_model"],    // chosen from declared dimensions
  "filters": [
    { "dimension": "gpu_model", "op": "=", "value": "H100" }
  ],
  "time": { "grain": "week", "range": "last_week" }
}

That structured request is handed to the same MetricFlow / Cube / Cortex Analyst engine every other consumer uses, which compiles it deterministically to the governed SQL — identical to the GMV compilation shown in ch. 08, applying the "rented ÷ listed GPU-hours" rule once, in one place. The model's output is validated before execution: is fill_rate a real metric? Is gpu_model a declared dimension? Is H100 a known value? If any check fails, the request is rejected and never becomes a wrong number.

RAG over the catalog, docs, and lineage

Grounding on the semantic layer answers the metric questions cleanly, but two gaps remain. First, the model still has to map messy human language onto the right metric and dimension — "fill rate," "occupancy," "how full are we" all point at fill_rate, and "the big Hoppers" means gpu_model = 'H100'. Second, plenty of legitimate questions ("which tables feed the churn model?", "what does dlperf_probe mean?") are about the platform itself, not a metric. Both are solved by retrieval-augmented generation: before the model answers, fetch the relevant context from the catalog, docs, and lineage (the assets of ch. 09 and ch. 10) and put it in the prompt.

The retrieved context does the disambiguation work. Metric descriptions ("Fill rate = rented GPU-hours ÷ listed GPU-hours, per gpu_model") tell the model which metric matches the intent. Column semantics resolve synonyms and acronyms. Lineage answers structural questions and lets an agent reason about blast radius. And a small set of prior vetted queries — real questions paired with their correct semantic-layer requests — act as few-shot examples that pin the model's output format and teach it the house vocabulary.

Agents for data tasks — beyond question answering

NL-to-SQL is the headline, but the same grounding-plus-tools pattern powers a broader class of agents that do platform work, not just answer questions. An agent is a model given tools (run a query, read the catalog, open a PR) and a loop to use them. The high-value GridDP agents are unglamorous and internal:

• Drafting dbt models — propose a staging or mart model from a source contract (ch. 01) and the modeling conventions of ch. 06. The agent drafts; a DPE reviews the PR.
• Documenting tables — generate column descriptions and a table summary from schema + sample + lineage, to backfill the catalog of ch. 09. Cheap, high-coverage, and a human skims before merge.
• Generating tests — propose dbt/data tests (not-null, uniqueness, the ledger-balances assertion from ch. 01) for a new model, extending the testing mindset of ch. 07.
• Triaging data incidents — when a freshness or quality check fires (ch. 10), an agent reads the lineage, identifies the upstream culprit, and posts a first-pass diagnosis to the on-call channel.

The accuracy stack — layered defense

No single technique makes an NL interface trustworthy; accuracy is a stack of defenses, each catching what the layer above missed. Like security, you assume any one layer can fail and design so a failure is caught, not shipped. From foundation to top:

1. Grounding on the semantic layer — the foundation. The model selects from governed metrics; the layer authors the SQL. Collapses the problem space.
2. Retrieval of catalog context — RAG feeds definitions, column semantics, lineage, and vetted examples so the selection is right.
3. Constrained generation / validation — the model's output must conform to the schema of valid metrics and dimensions; an unknown metric or dimension is rejected before any query runs. This is where most hallucinations die cheaply.
4. Execution + self-correction — run the compiled query; if the engine returns an error (a type mismatch, an empty result where one is impossible), feed the error back and let the agent retry once or twice. Bounded loops only — a self-correction loop with no cap is a cost and latency hazard.
5. Confidence + abstention — the last line. If the model can't map the question to a known metric, or retrieval comes back thin, or the question is genuinely out of scope, the correct output is "I'm not sure — here's what I can answer", not a guess. Designed abstention is the single feature that separates a trustworthy assistant from a misinformation pipeline.

Evaluating NL-to-SQL — the testing mindset, applied to AI

You cannot manage accuracy you do not measure, and "it looked right in the demo" is not measurement. The discipline is the same one ch. 07 brought to transformations: a curated eval set of question → known-correct-answer pairs, run as a regression test on every change to the prompt, the retrieval, or the model. The metric that matters is execution accuracy — does the result the system returns match the ground-truth result? — not whether the generated query string looks plausible, since two very different queries can return the same correct number and two similar ones can diverge.

Two numbers are tracked together, because they trade off: execution accuracy (of the questions it answered, how many were right) and abstention rate (how often it correctly said "I'm not sure" versus refusing answerable questions). A system that answers everything with 80% accuracy is worse than one that answers 70% of questions at 99% accuracy and abstains on the rest — because the second one never lies. Every prompt tweak or model swap reruns the suite, and a regression blocks the change, exactly like a failing dbt test blocks a merge.

- id: fill_rate_h100_lastweek
  question: "what was H100 fill rate last week?"
  expect:
    metric: fill_rate                 # must resolve to the catalog metric
    dimensions: [gpu_model]
    filters: [{ gpu_model: "H100" }]
    time: { grain: week, range: last_week }
  ground_truth_result: 0.683          # reconciled offline by an analyst
  tolerance: 0.005                     # exec-accuracy: |result - truth| <= tol
  must_not_abstain: true               # this IS answerable; abstaining = fail

- id: ceo_favorite_color
  question: "what is the CEO's favorite color?"
  expect: { abstain: true }            # out of scope → MUST abstain, not guess

The eval set is a living asset: every real-world miss becomes a new test case, so the system can never regress on a question it once got wrong. This is the same flywheel as data quality in ch. 10 — observe a failure, encode it as a check, prevent its recurrence — applied to the AI layer instead of the pipeline.

Guardrails — what the AI is allowed to touch

Accuracy is about returning the right answer; guardrails are about bounding what the system can do in the process. Even a perfectly grounded agent must run inside the access-control and privacy fence from ch. 11 — the AI is a consumer, and consumers get scoped permissions, not a master key.

The unifying principle from ch. 11 holds: the AI gets least privilege. It authenticates as a specific, narrowly-scoped role; it can only reach the domains its channel is cleared for; and everything it does is audited identically to a human analyst. Grounding makes the answers correct; guardrails make sure that even a wrong or manipulated request can't do damage beyond returning a number.

Worked end-to-end: "what was H100 fill rate last week?"

Tie the whole stack together on the question that opened the chapter. A supply-ops engineer types it into Slack; here is every step, and why each is correct by construction rather than by luck.

The compiled query is the governed one from the ch. 08 catalog — rented GPU-hours over listed GPU-hours, per gpu_model — running against the fct_telemetry_hourly rollup, not the raw firehose. Walk the contrast with the naive path:

• The model never chose a join or a filter, so it could not skip instance or forget the settled-only rule — the failure modes from the table simply have no surface to occur on.
• The answer is the one company-agreed fill rate, identical to what the supply-ops dashboard shows, because both compile through the same semantic-layer definition. There is no second number to disagree with.
• The reply states its own definition and links the query — so the human can verify, and trust is earned rather than assumed.
• Ask it something out of scope ("CEO's favorite color") or gated ("show me flagged-fraud accounts") and it abstains or refuses, because of the accuracy stack's top layer and the guardrails.

Takeaway

• The AI is GridDP's newest and riskiest consumer: the promise is English questions answered in Slack; the peril is a confidently wrong number, which is worse than no number. "Just point an LLM at the warehouse" reproduces the five-way definition chaos of ch. 08, one query at a time.
• Naive text-to-SQL on raw schemas fails on ambiguous columns, invented joins, unseen business logic, and multiple metric definitions — missing-context failures a bigger model does not fix.
• Grounding on the semantic layer is the accuracy lever: the model picks metrics and dimensions from the governed menu and the layer compiles the one correct SQL, collapsing "write any SQL" into "pick from validated correctness." This is why ch. 08 is the prerequisite, not a footnote.
• Around that foundation: RAG over the catalog/lineage for disambiguation, an accuracy stack (grounding → retrieval → validation → self-correction → abstention), execution-accuracy evals that block regressions, and ch. 11 guardrails (read-only, cost caps, PII filtering, audit) that bound what the agent can touch.
• Accuracy is a platform property, engineered from grounding + governance + evals — not a model property bought with more parameters.

The grounded NL interface now leans hard on the telemetry rollups and the warehouse it queries under cost caps — which raises the question every prior chapter deferred: how does all of this stay fast and affordable at 150 M samples a day? Next: performance & scale.