Serving Architectures & Training/Serving Skew

Three ways a prediction reaches the product

Every ML serving question starts with one decision that is usually made by accident: when is the prediction computed relative to when it is used? Compute it ahead of time for every entity and you have batch scoring — cheap, simple, and stale by construction. Compute it on request and you have online inference — fresh, but now every millisecond of feature retrieval is on the product's critical path. Compute it as events arrive and you have streaming inference — the prediction is ready before anyone asks, at the cost of a stateful pipeline that never stops.

GridDP needs all three. Churn scores are batch — nobody needs a churn probability faster than the daily retention campaign that consumes it. Ranking is online — the offers must be scored for this search, with these filters, inside the page load. Fraud is streaming-then-online — velocity features stream continuously, the score is computed on the signup request. Capacity forecasts are batch with an hourly cadence; dynamic pricing is streaming, republished every minute per market. Matching the path to the decision is the first job; the second is keeping the features the model sees at serving time identical to the ones it trained on.

The three data paths, compared

The latency budget

Online inference lives inside a budget set by the product, and the model gets a slice of it. GridDP's search page allows ~50 ms server-side for ranking. A realistic split: 5 ms candidate retrieval (the ~80 offers that match the filters), 10 ms feature retrieval, 20 ms model inference across the candidate set, 5 ms post-processing and logging, 10 ms headroom for p99. Ten milliseconds of feature retrieval means one round trip to a store that answers in single-digit milliseconds, with all features for all candidates fetched in a batch — not a query per feature, and certainly not a query to the lakehouse.

• Batch the reads. One multi-get for 80 offer keys × the feature vector each, not 80 × 12 single reads. Store all of an entity's features in one value (a hash or a serialized row) so the read count equals the entity count.
• Precompute joins. If the ranking model needs host features for each offer, materialize them onto the offer key at write time. Joins at request time are the budget's enemy.
• Bound the candidate set. Score 80 offers, not 8,000. Candidate generation (filters, a cheap first-stage model) is a data-architecture decision as much as a modeling one.
• Measure p99, budget for it. A store with 2 ms p50 and 40 ms p99 blows the budget one request in a hundred, which at search volume is constantly.

Choosing the online store

The online store holds the latest feature values per entity and answers key lookups fast. Everything else — durability, TTLs, multi-region, cost per read — is what distinguishes the options.

The rule: Redis when the working set fits in memory and the budget is single-digit milliseconds (ranking, fraud); DynamoDB or Bigtable when the keyspace is large and durable and 5–10 ms is fine (per-account features for a product surface with a looser budget); the managed online table when the platform team is small and the latency ceiling is acceptable. Whichever it is, the materialization from the offline store must be the same job that builds training features — that is where consistency is won.

Input assembly & caching

Between the store and the model sits input assembly: fetch precomputed features, compute on-demand features from the request, apply the same transformations the training pipeline applied (scaling, encoding, imputation), and lay it all out in the column order the model expects. This code path is where skew is born, so treat it as data infrastructure, not glue: the transformation code should be the same artifact the training pipeline used (a versioned preprocessing module shipped with the model), and the assembled vector should be logged.

def score_search(request, candidates):                      # ~80 offers
    keys = [f"offer:{c.offer_id}" for c in candidates]
    precomputed = online_store.mget(keys)                    # ONE round trip; hash per offer holds host + offer feats

    rows = []
    for c, feats in zip(candidates, precomputed):
        feats = feats or DEFAULTS                            # missing key → training-time defaults, never None
        feats["price_vs_search_max"] = c.price_hr / request.max_price   # on-demand: depends on the request
        feats["region_match"]        = int(c.region == request.region)
        rows.append(feats)

    X = preprocess(rows, spec=model.preprocess_spec)         # SAME versioned transform used in training
    scores = model.predict(X)

    feature_log.emit(request.id, model.version, rows, scores)   # log what the model actually saw
    return rank(candidates, scores)

Caching helps in two places and hurts in one. Cache assembled feature vectors for hot entities (the same 500 popular offers appear in most searches) with a TTL shorter than the feature freshness SLA. Cache model outputs for identical (entity, context) pairs when the context is coarse. Do not cache across a materialization boundary without invalidation — a cached vector that outlives the nightly refresh is a stale feature with a fast read time.

Training/serving skew: the four sources

Skew is any difference between the feature values a model saw in training and the values it sees in serving. It is the most common reason a model that evaluated well performs badly, and it comes from four distinct places, each with a distinct fix.

1. Definition skew. The feature is computed by different code offline and online — SQL in the training pipeline, Python in the service, a Flink job for the streamed version. Values disagree in edge cases: null handling, time-zone, window boundaries, integer division. The second-definition problem from chapter 06 is this.
2. Freshness skew. Training used point-in-time-correct values (the feature as of the prediction time); serving reads the online store, which was materialized N hours ago. The model learned on values that were, on average, fresher or staler than what it sees live.
3. Transformation skew. Scaling parameters, category encodings, imputation defaults computed on the training set and re-implemented — or recomputed — at serving. A new category ("B200") encodes as unknown offline and as a fresh index online.
4. Leakage-induced skew. The training set contained information unavailable at serving time (a future label, a post-decision field). Offline the model leaned on it; online the field is missing or default. Strictly a training-data bug (chapter 02), but it presents as serving skew.

Killing skew

• Log served features; train on the log. The single highest-leverage practice. The inference service records the assembled feature vector (and model version) for every scored request. Training sets are built by joining labels to these logs, not by recomputing features from the warehouse. Definition, freshness and transformation skew all vanish, because training data is serving data. Cost: log volume (sample if needed) and no backfill for brand-new features (bootstrap those from the offline store, then switch).
• One transform artifact. Preprocessing is a versioned module packaged with the model; the training pipeline and the serving path import the same one. Encodings and scaling parameters are fitted once and serialized into it.
• Materialize from the same definition. The offline feature table and the online store are written by the same feature pipeline; the online value is the latest row of the offline history, nothing else.
• Skew monitoring. Daily, for a sample of served requests: recompute the features from the offline store as of the request time, compare to the logged served values, report per-feature disagreement rate and magnitude. Alert at a tolerance. This catches definition and freshness skew within a day.
• Shadow before switch. A new model or feature pipeline runs in shadow, logging its would-be outputs against the live one; disagreement beyond expectation blocks the cut-over.

Worked example: GridDP ranking

Ranking is GridDP's hardest serving problem because it is online, high-QPS and scores a candidate set. The design: offer and host features are materialized nightly and hourly into Redis hashes keyed by offer_id (host features denormalized onto the offer at write time, so a search needs one multi-get); a streamed last_heartbeat_age gates availability; two on-demand features come from the request. The service logs every assembled vector at 100% sampling to a Kafka topic that lands in Iceberg; the ranking training set is built from that log joined to click and rental outcomes. Preprocessing ships inside the MLflow model artifact. A daily skew job recomputes a 1% sample from the offline store and reports disagreement per feature — the tolerance is 1% of rows for any feature, and the day it fired was a Redis materialization job that silently failed for one region, leaving stale price_vs_market values: caught in 20 hours instead of at the next quarterly model review.

When to choose what

Next: how you prove, a year later, exactly which data and features produced a model — 09 · Lineage, Metadata & Reproducibility.

✓ Check yourself

Answer from memory before opening each one.

1. A product manager wants live churn probabilities on a dashboard. Before you design an online inference service, what question do you ask, and why does the answer remove a whole class of problems?

   Show answer

   Ask whether the prediction can be an hour old. If it can, score everyone in batch against the offline store and serve a lookup. The model then runs on the same data it trained on, so training and serving skew cannot occur, and the serving path has no model in it. See three ways a prediction reaches the product.

2. Ranking has 50 ms server-side and must score 80 candidate offers. A store has a 2 ms median and a 40 ms p99. What happens, and what read pattern fits the budget?

   Show answer

   About one request in a hundred blows the budget, which at search volume happens constantly. Read once with a multi-get for all 80 keys, keep an entity's features in one value, precompute joins at write time and bound the candidate set. See the latency budget.

3. The service reads a key that is missing from the online store. One engineer wants to pass null, another wants defaults. Which is right, and what must the defaults match?

   Show answer

   Use defaults, but the training-time defaults, so the model sees what it saw in training for an unknown entity, and log that the default was used. A silent null can land in a bucket the model reads as low-risk. See input assembly and caching.

4. Offline evaluation says 0.94 and production behaves like 0.85. List the four sources of skew and the order you would check them in a service that recomputes features in Python.

   Show answer

   Definition skew comes first, since the service has its own implementation of the feature. Then transformation skew, from re-implemented scaling or encoding, then freshness skew, from a store materialised hours ago, and last leakage that only looks like serving skew. See the four sources of skew.

5. Your service logs the assembled feature vector for every scored request. Which skew sources does training on that log remove, and what does it cost?

   Show answer

   It removes definition, freshness and transformation skew, because the training data is the serving data. It costs log volume, which you can sample, and it cannot backfill a brand-new feature, which needs a bootstrap from the offline pipeline. See killing skew.

6. Choose an online store for fraud, with a single-digit millisecond budget, and for a per-account surface with a 50 ms budget, and say what stays the same.

   Show answer

   Redis suits fraud when the working set fits in memory, since p99 reads are around a millisecond or two. DynamoDB or Bigtable suit large, durable keyspaces where 5 to 10 ms is fine. In both cases the store is filled by the same pipeline that builds training features. See choosing the online store.

Takeaway

• Decide when the prediction is computed before anything else, and push every prediction that can be an hour old into batch scoring, where most skew cannot occur.
• Online inference lives in a latency budget, so batch the reads, precompute joins, bound the candidate set and budget for p99, not the median.
• Choose the online store by working-set size and latency: Redis for in-memory single-digit milliseconds, DynamoDB or Bigtable for large durable keyspaces.
• Input assembly is data infrastructure, so ship one versioned preprocessing artifact to training and serving and use training-time defaults for missing keys.
• Skew has four sources, definition, freshness, transformation and leakage, and it raises no error, so it must be measured.
• Log the served feature vector and train on it, monitor skew daily against a tolerance, and run new pipelines in shadow before switching.

Logging what the model saw is one half of reproducibility; the other half is tracing every artifact back to its inputs. → Lineage, Metadata & Reproducibility

Further reading

• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — Breck et al., Google, 2017. Concrete tests for data, model and serving, including skew checks.
• Rules of Machine Learning: Best Practices for ML Engineering — Google, accessed 2026. Rules on training and serving skew and on logging features at serving time.
• The Tail at Scale — Dean and Barroso, Google, 2013. Why p99 latency, not the median, sets the budget for a fan-out request.
• Feast architecture overview — Feast documentation, accessed 2026. How online retrieval and materialisation are structured in an open-source feature store.
• MGET — Redis documentation, accessed 2026. The multi-key read behind the single round trip in the latency budget.