Feature Engineering at Scale & the Feature Store

Why feature stores exist

By the time GridDP had four models, it had four copies of host_reliability_7d: a dbt model for the forecaster, a pandas function in the ranking notebook, a Flink job for fraud, and a hand-written SQL string in the pricing service. They disagreed on the window boundary, on whether preempted rentals counted, and on what to do when a host had no rentals. Every model was trained on one definition and served with another. That is the problem a feature store solves, and it is worth being precise about how: a feature store is a registry of feature definitions, plus an offline store that keeps their history, plus an online store that keeps their latest values, with the guarantee that all three are produced from the same definition. Everything else — UIs, SDKs, monitoring — is convenience.

Anatomy: registry, offline store, online store

  REGISTRY (the source of truth — code in git, or a metadata service)
  ┌──────────────────────────────────────────────────────────────────────┐
  │ entity  host  (key: host_id)                                          │
  │ feature host_reliability_7d : float                                   │
  │   source: gold.fct_rental   window: 7d   agg: 1 - preempted/total      │
  │   owner: supply-ml   ttl: 2d   version: 3   tests: not_null, in[0,1]   │
  └──────────────────────────────────────────────────────────────────────┘
        │ one definition drives both pipelines
        ├──────────────────────────────┬───────────────────────────────────┐
        ▼                              ▼                                   ▼
  BATCH pipeline                 STREAMING pipeline                ON-DEMAND (request-time)
  dbt / Spark, hourly            Flink / Spark SS, seconds         pure function of request +
  → offline store, full          → online store directly           stored features, e.g.
    history (valid_from/to)        (and appended to offline)          price_vs_market = req.price
        │                              │                                / market_median_1h
        ▼                              ▼
  ┌──────────────────┐  materialize  ┌──────────────────┐
  │ OFFLINE STORE     │ ───────────▶ │ ONLINE STORE      │──▶ get_online_features(host_id)
  │ Iceberg / Delta   │  latest per  │ Redis / DynamoDB  │    < 10 ms, TTL-expired = null
  │ point-in-time     │  entity      │ latest value only │
  └──────────────────┘              └──────────────────┘
        │
        ▼ get_historical_features(spine) → training set (ch. 02)

Entities, keys and feature views

Features attach to entities — the things you make predictions about — via a key. GridDP's entities are host, machine, gpu_model × region, account, payment, listing, and composite ones like (account, gpu_model). Get the entity model wrong and everything downstream fights it: a feature keyed on machine_id cannot be served for a listing until you know the mapping at request time, and that mapping had better be in the online store too.

A feature view (Feast's term; Tecton says feature view, Hopsworks says feature group, Databricks says feature table) groups features that share an entity, a source and a cadence — host_reliability_features holding host_reliability_7d, host_preempt_rate_30d, host_uptime_90d. Views are the unit of materialisation and of ownership. Keep them cohesive: one view per (entity, source, cadence), not one giant view per model.

host = Entity(name="host", join_keys=["host_id"])

host_reliability = FeatureView(
    name="host_reliability_features",
    entities=[host],
    ttl=timedelta(days=2),                     # online value older than this → null, not stale
    schema=[Field("host_reliability_7d", Float32),
            Field("host_preempt_rate_30d", Float32),
            Field("host_uptime_90d", Float32)],
    source=BatchSource(table="gold.host_reliability_daily",
                       timestamp_field="valid_from",      # availability time (ch. 02)
                       created_timestamp_column="materialized_at"),
    owner="supply-ml@griddp", tags={"tier": "1"},
)

# training:  fs.get_historical_features(entity_df=spine, features=[...])   → point-in-time join
# serving:   fs.get_online_features(entity_rows=[{"host_id": "h_9b2"}], features=[...])

The ttl line is doing quiet, important work: it defines how stale a served value may be before the store returns null rather than a value from three days ago. Pick it per view from the feature's cadence plus tolerance — and make sure the model's null handling is deliberate (ch. 08).

Point-in-time retrieval and backfills

The offline store is interval-keyed: each row is (entity, valid_from, values…), and valid_to is implied by the next row. get_historical_features is the ASOF join of chapter 02, executed by the store against a spine you provide. Two operations keep the history correct:

• Backfill — computing a new or changed feature for the past, so models can train on it. A backfill must produce values as they would have been computed at each historical time, which means running the definition over historical source snapshots, not over today's tables. For definitions with late-arriving data, the backfilled value can legitimately differ from what was served at the time; record materialized_at so the training join can choose. Backfills are the most expensive operation in the store — a 7-day window over 18 months of the telemetry rollups is a full-table scan per day of history — so they are batched, incremental (only the changed feature), and scheduled off-peak.
• Materialisation — pushing the latest offline values into the online store. Batch features materialise on the pipeline's cadence; the interval between the offline value existing and the online store having it is exactly the availability gap from chapter 02, so the store must record when each key was written.

Streaming and on-demand features

Batch views cover anything with a daily-or-hourly cadence. Two other kinds cover the rest:

Streaming features are where a store proves its worth or its cost. The definition must be executable by both a batch engine (for backfills and training) and a stream engine (for serving). Products that own the transformation (Tecton, Hopsworks, Databricks with Spark Structured Streaming) can compile one definition to both; a registry-only product (Feast) leaves that consistency to you.

Build vs buy: the products, honestly

Worked example: GridDP's feature inventory

After consolidating the four copies, GridDP's registry has eleven feature views across six entities; roughly 60% of feature usage comes from four views (host_reliability_features, account_activity_features, market_depth_features, telemetry_rollup_features) — the reuse that justified the store. On Stack A: definitions in a feature_repo/ Git repo (Feast), batch views materialised by dbt + Dagster into Iceberg and pushed to DynamoDB hourly, three streaming views written by Flink to DynamoDB and appended to Iceberg, on-demand features as registered Python functions used by both the fraud API and the dataset builder. Online p99 is 6 ms for a 30-feature vector across three views; the fraud request budget is 100 ms, so 6 is comfortable. The costliest line item is not the store — it is the monthly backfill when a definition changes, which reads 14 TB of rollups. That number drove the partition layout in chapter 05.

When you need a feature store — and when you don't

The mature answer names the flip condition: "Today we have one nightly model; dbt tables and an ASOF join are the feature store. The day fraud needs acct_payment_velocity_1h in the request path, I introduce a registry and an online store — and if a second streaming feature follows, I'd rather buy the stream-batch consistency than build it."

All of this history has to live somewhere, and be versionable: 05 · Storage for ML: Lakehouse, Formats & Dataset Versioning.

✓ Check yourself

Answer from memory before opening each one.

1. GridDP has four copies of host_reliability_7d in dbt, pandas, Flink and a SQL string. A manager proposes a weekly meeting to keep them aligned. Why does that not solve the problem, and what does?

   Show answer

   Each copy is a separate implementation with its own window boundary and null handling, so they drift between meetings and a model trains on one while serving another. Define the feature once in a registry and compute both paths from that definition, which turns agreement into a property of the system. See why feature stores exist.

2. A feature view has a TTL of two days, the batch job fails for three days, and the model serves host reliability for a host. What does the API receive, and what must the model do about it?

   Show answer

   The online store returns null, not the three-day-old value, because the TTL marks it as too stale. The model's null handling must be deliberate, for example a documented fallback or a skipped feature, since an imputer that maps null to a low-risk bucket hides the outage. See entities, keys and feature views.

3. You change host_reliability_7d to exclude maintenance windows. Why is that a new version and not an edit, and what breaks if you edit in place?

   Show answer

   Training sets and models reference the old definition, so rewriting history invalidates every model trained on it and every skew comparison against it. Write version 3 beside version 2 and let datasets pin the version they used. See point-in-time retrieval and backfills.

4. A backfill of a seven-day telemetry window over 18 months is queued for tomorrow. Which source do you run it over, and why might the backfilled value differ from what was served at the time?

   Show answer

   Run it over historical source snapshots as they stood at each time, not over today's tables. Late-arriving data means the recomputed value can legitimately differ from the served one, so record materialized_at and let the training join choose. See point-in-time retrieval and backfills.

5. Your team has a registry-only tool such as Feast and adds a streaming velocity feature. Where does the consistency risk now sit, and what would change if you bought a product that owns computation?

   Show answer

   The registry holds the definition, but you own the batch and stream implementations, so you must keep their window semantics identical and make the offline history the stream's output. A product that compiles one definition to both engines moves that guarantee into the product. See streaming and on-demand features.

6. The company has one nightly forecaster and a CTO who wants a feature store this quarter. Argue for and against, and name the event that flips your answer.

   Show answer

   Against: the models are batch, features come from the same dbt tables as training data, and versioned tables with an as-of join and a definitions document are the store. For: a registry pays off once a request-path model needs fresh features. Flip when fraud needs a velocity feature in the request path, and again when a second streaming feature follows. See when you need a feature store.

Takeaway

• A feature store is a contract, define once and serve twice, and a team with the contract and no product has one while a team with the product and duplicated definitions does not.
• A registry, an offline store with validity intervals and an online store with latest values form one system, and the registry is the source of truth.
• Entity model and feature views come first: group features by entity, source and cadence, and set a TTL that makes stale values null instead of old.
• A definition change is a new version, and a backfill must run over historical snapshots so training sees values as they were known.
• Streaming and on-demand features are where the store earns its cost, and the offline history must come from the same logic as the online value.
• Choose build or buy on the consistency guarantee and team shape: batch-only models need tables and an as-of join, and shared streaming features justify a product.

A feature store needs storage that keeps history and can pin versions. → Storage for ML: Lakehouse, Formats & Dataset Versioning

Further reading

• Feast architecture overview — Feast documentation, accessed 2026. Registry, offline store, online store and materialisation in an open-source feature store.
• Point-in-time joins — Feast documentation, accessed 2026. How historical retrieval picks the latest value as of each row's timestamp, and how TTL limits it.
• Tecton documentation — Tecton, accessed 2026. How a commercial platform compiles one definition to batch, streaming and on-demand execution.
• Hidden Technical Debt in Machine Learning Systems — Sculley et al., NeurIPS 2015. Why duplicated data dependencies and glue code become the main cost.