The Platform for AI/ML

GridDP's ML use cases — grounded in data we already modeled

A platform-for-ML chapter that opens with "what is a feature store" in the abstract teaches nothing. So we start where a DPE actually starts: with the models the business wants to ship, and the tables those models pull from. Every feature below already lives somewhere in GridDP's source map (ch. 01) and its dimensional model (ch. 06). The DPE's job is not to invent features — it is to make the data that already exists reachable, correct, and reproducible for training and serving.

GridDP has five models in or near production, and they are deliberately varied — a ranking model, a classifier, a churn predictor, a forecaster, and a market-pricing model — because each stresses the platform differently.

The feature store: why it exists

Notice the overlap in the table above. reliability score feeds ranking and churn. hourly utilization rollups feed forecasting and ranking. bid depth feeds pricing and fraud. Without a shared layer, each team re-implements these computations in their own training notebook, slightly differently, and then re-implements them again in production serving code — three forks of "reliability score" that quietly disagree. The feature store exists to solve two problems at once:

• Reuse. Define a feature once (host_reliability_7d), register it, and every model — and every engineer — consumes the same definition.
• Train/serve consistency. Serve the same feature values offline (for training, as a point-in-time history) and online (for inference, at low latency). This is the deep problem we return to in offline↔online parity.

Architecturally a feature store is three components — an offline store (history, in the lakehouse), an online store (latest values, in a low-latency KV), and a registry (the definitions, the source of truth for what a feature means) — fed by feature pipelines that can be batch or streaming.

Training-data pipelines & point-in-time correctness

Building a training set sounds trivial — join labels to features — and that one join is where most production ML quietly breaks. The pitfall has a name: label leakage, the use of information that would not have been available at prediction time. A churn model that "knows" the account already closed; a fraud model that includes the chargeback it is supposed to predict; a ranking model trained on a reliability score computed after the rental completed. Every one inflates offline metrics and collapses in production.

The defense is point-in-time correctness: for each training row, you may only use feature values that were known as of the prediction timestamp. Concretely — to build a churn label for account a_88 with a prediction time of 2026-04-01, you join the features that existed on 2026-04-01, never the value the feature held on 2026-05-01, even though that newer value sits in the same table today.

The mechanism that makes this possible is something we built three chapters earlier: slowly-changing dimensions (SCD2) from ch. 06. Because the offline store keeps every version of a feature with its valid-from / valid-to interval, a point-in-time join can ask "what was the value as of the prediction time" rather than "what is the value now." A feature store's offline join is, under the hood, an AS OF join against SCD2-style history.

-- For each (account, prediction_ts) we want the LAST feature value
-- whose validity started at or before prediction_ts. No future leakage.
WITH labels AS (
  SELECT account_id,
         prediction_ts,
         churned_30d            -- label: measured in the 30d AFTER prediction_ts
  FROM   ml.churn_labels
)
SELECT
  l.account_id,
  l.prediction_ts,
  l.churned_30d,
  f_rel.value  AS host_reliability_7d,
  f_spend.value AS acct_spend_trend_30d
FROM labels l
-- ASOF join: pick the feature row valid AS OF the prediction timestamp
ASOF JOIN feature_store.host_reliability_7d f_rel
       MATCH_CONDITION (l.prediction_ts >= f_rel.valid_from)   -- only past values
       ON l.account_id = f_rel.account_id
ASOF JOIN feature_store.acct_spend_trend_30d f_spend
       MATCH_CONDITION (l.prediction_ts >= f_spend.valid_from)
       ON l.account_id = f_spend.account_id;

Embeddings & vector storage

Not every feature is a number in a table. Several GridDP problems are fundamentally about similarity, and similarity wants embeddings — dense vectors where "close" means "alike":

• Marketplace search & similarity — "find offers like this one" or semantic matching of a free-text workload description ("fine-tune a 70B model") to suitable GPU configurations.
• Host clustering — embedding a host's telemetry-behavior signature to find machines that behave alike (useful for both reliability scoring and fraud rings).
• RAG over internal data — retrieval for the AI interfaces in ch. 13 (docs, runbooks, metric definitions). The same vector infrastructure serves the platform-for-AI and the AI-for-platform sides.

An embedding pipeline is just another feature pipeline: a model (hosted or API) turns rows or documents into vectors on a schedule or a stream, and the vectors land in a store with an approximate-nearest-neighbour (ANN) index. Where they land is the decision:

• pgvector — a Postgres extension. Right when volumes are modest and you already run Postgres; co-locates vectors with relational filters.
• Warehouse-native vector types — Snowflake VECTOR, Databricks Vector Search, BigQuery vectors. Right when the embeddings live next to the rest of your governed data and you want one access-control plane.
• Dedicated vector DBs — Milvus, Qdrant, Weaviate, Pinecone. Right at high vector volume / high QPS where ANN index quality and recall tuning are first-class concerns.

Reproducibility: versioning data, features, and models together

The question a regulator, an incident review, or a "why did ranking change last Tuesday?" investigation all ask is the same: can you retrain this exact model on the exact data it originally saw? If the answer is no, you have a model you cannot debug, audit, or roll back. Reproducibility requires versioning three things in lockstep:

• Data — pin the training set to immutable table snapshots. Iceberg/Delta time-travel (a snapshot ID or commit version) makes "the data as of this commit" a first-class, queryable thing.
• Features — pin the feature definitions (the registry version) and the materialization, so host_reliability_7d v3 is unambiguous.
• Model — pin the trained artifact, hyperparameters, code version, and metrics in a model registry (MLflow).

Tie all three together and you get the chain that matters: model run → feature versions → table snapshots → source tables. That is the same lineage graph from ch. 10, extended past the warehouse into the model. When ranking degrades, lineage lets you walk from the deployed model back to the precise telemetry rollups it trained on — and discover, say, that a host-agent schema change three weeks ago silently shifted a feature distribution.

Online feature serving

Training tolerates minutes; inference does not. When a renter loads search results, the ranking model must score dozens of offers inside the page's latency budget — call it ~50 ms end-to-end, of which feature retrieval gets maybe 10 ms. You cannot run a point-in-time join over the lakehouse in 10 ms. This is the entire reason the online store exists: it holds the latest value of each feature, keyed by entity, for single-digit-millisecond lookups.

The design tension is precompute vs on-demand:

• Precomputed (materialized) features — batch/streaming pipelines write the latest values to the online store ahead of time. Cheap and fast at read time; the lookup is a single key fetch. Use for anything that doesn't depend on request context: host_reliability_7d, acct_rentals_30d, DLPerf scores.
• On-demand features — computed at request time from the request payload (e.g. price-vs-this-search's-filters, distance from the renter's requested region). Necessary when the value can't be precomputed, but every on-demand computation eats the latency budget.

The rule of thumb: precompute everything you can, compute on-demand only what depends on the live request. Fraud scoring leans more on streaming-fresh features (device/IP velocity in the last five minutes) — which is why the streaming feature pipeline writes straight to the online store, not just to the offline lakehouse.

ML observability: drift, decay, and the feedback loop

A deployed model is a data pipeline whose output happens to be predictions — and like any pipeline it degrades silently. ML observability is monitoring built as a data pipeline over four things:

• Feature drift — the distribution of an input feature shifts vs training (a new GPU model floods the fleet; price_vs_market re-centres). The model is now extrapolating.
• Prediction drift — the distribution of the model's output shifts (fraud scores creep up across the board). Often the first visible symptom.
• Performance decay — measured accuracy/AUC falls as ground-truth labels arrive. Lags reality, because labels (did the account actually churn?) take 30 days to land.
• The feedback loop — the dangerous, marketplace-specific one. A new ranking model demotes a class of hosts → those hosts get fewer rentals → their measured utilization drops → which the reliability feature reads as degradation → which demotes them further. The model's own deployment corrupts its future inputs.

Mechanically this is the same machinery as ch. 10 data quality, pointed at features and predictions: log every prediction with its feature vector and model version, compare distributions on a schedule (PSI, KL-divergence), and alert on drift. Because predictions are logged with their inputs, the feedback loop is detectable — you can correlate a ranking deployment with a subsequent shift in fleet-level reliability and catch the doom loop before it compounds.

Offline↔online parity (training/serving skew)

This is the bug the feature store was invented to kill, and it deserves its own section. Training/serving skew is when the features a model trains on differ from the features it sees in production — even subtly — so a model that validated beautifully offline makes worse, weirder predictions live. The causes are mundane and ubiquitous:

• Two implementations. The training feature is computed in a dbt/Spark batch job; the serving feature is re-implemented in the inference service's application code. They round differently, default nulls differently, define "last 7 days" differently. Two code paths, one definition — they will diverge.
• Time semantics. Offline you (correctly) take the value as of prediction time; online you take "now." If the offline join wasn't point-in-time-correct (above), training saw future-tinged values production never has.
• Freshness gaps. Training uses a fully-backfilled feature; production reads a stale online value because the materialization job lagged.

The feature store's structural answer: a feature is defined once in the registry, the same computed values are written to both the offline store (for training) and the online store (for serving), and both training and inference call the same retrieval API. Skew doesn't get "reduced" — it's eliminated by construction, because there is only one definition and one set of values. Streaming features get materialized to both stores from the same pipeline for the same reason.

The ML / feature platform per stack

Same three-component shape (offline store, online store, registry) across all three reference stacks; what changes is integration, governance, and lock-in. This is the row "ML / features" from Start Here, expanded.

The trade-off mirrors ch. 03: A is maximally composable and lock-in-free but you wire the offline/online/registry pieces together yourself; B is the strongest if Spark-scale ML and a unified governance plane (Unity Catalog) are central; C wins when your ML lives close to SQL-modeled data and you want the warehouse to be the feature store too. A marketplace at GridDP's scale typically runs the warehouse-native feature store for tabular models and bolts on a dedicated vector store only when a genuine similarity workload appears.

Takeaway

• GridDP's five models — ranking, fraud, churn, capacity forecasting, dynamic pricing — all pull from data already modeled in ch. 06; the DPE's job is to make it reachable, correct, and reproducible, not to invent features.
• The feature store earns its keep by giving you reuse (one definition, many models) and train/serve consistency (same values offline and online), via three components: offline store, online store, registry.
• Point-in-time correctness — powered by SCD2 history from ch. 06 — is what prevents label leakage, the most expensive and most invisible bug in applied ML.
• Reach for embeddings/vector stores only for genuine similarity/unstructured problems; most tabular models don't need them.
• Reproducibility means versioning data, features, and models together, with lineage that extends from ch. 10 all the way into the model run.
• Online serving lives or dies on latency budgets: precompute everything you can, compute on-demand only what depends on the live request.
• ML observability is monitoring-as-a-pipeline over feature drift, prediction drift, and decay — plus the marketplace-specific feedback loop only the platform can see.
• Offline↔online parity is the platform's highest-leverage gift to ML: eliminate training/serving skew by construction with one definition and a dual write.

We have built the platform for the company's models. Next we flip the arrow: how AI is increasingly the consumer and operator of the platform itself — the semantic layer, NL-to-SQL, and agents. → AI powering the platform