Streaming Features & Real-Time Inference Data

Which features actually need to stream

Streaming is the most over-prescribed component in ML data architecture. The reflex is "real-time inference needs real-time features", and it is wrong for most features: a ranking model scoring offers in 50 ms does not need host_reliability_7d recomputed since the last heartbeat — a value materialized an hour ago is indistinguishable from a live one. What it needs is fast reads, which is the online store's job (chapter 08), not a streaming pipeline's.

A feature earns a streaming pipeline when the value changes materially inside the batch interval and the model's decision depends on that change. At GridDP that is a short list: device/IP signup velocity in the last five minutes (fraud), live bid depth per GPU model (pricing), a host's last-heartbeat age (ranking's availability gate), and an account's actions in the current session (fraud, churn). Everything else — tenure, spend trends, benchmark scores, 30-day aggregates — is batch, and forcing it through Flink buys you cost and a second definition of the same feature to keep in sync.

Freshness SLAs, in numbers

Write the SLA down before choosing the engine, because the engine follows from it. Freshness is measured end to end — event occurs → feature value visible to the model — and it has three components: transport lag (producer to Kafka), compute lag (window close plus processing), and write lag (to the online store). A "sub-second feature" that sits behind a five-minute tumbling window is a five-minute feature.

The table has a shape worth noticing: cost climbs a step per band, and the top band is where you stop having one feature definition. That is the real price of streaming, and it is paid in consistency, not in cloud spend.

Computing features on a stream

Three engine families cover nearly every streaming-feature pipeline. The choice is about state and latency, not brand.

• Apache Flink. True event-at-a-time processing with keyed state, event-time windows, watermarks and exactly-once sinks. The right tool when latency is seconds, windows are sliding or session-shaped, and state (per-account counters, per-IP sets) must survive restarts. Flink SQL makes many feature definitions declarative.
• Spark Structured Streaming. Micro-batches on a trigger; same API as the batch job, which is its superpower — the hourly feature and the two-minute feature can share code. Latency floors at the trigger interval; state handling is coarser than Flink's. The default in Stack B.
• Kafka Streams / ksqlDB. Library-in-your-service model, no cluster to run. Good for a fraud service that owns its own velocity counters; awkward when the feature must also land in the lakehouse for training.

Managed offerings (Confluent Flink, Databricks streaming tables, Snowflake Dynamic Tables at minute lag, Tecton's streaming feature views) shift the operations, not the semantics. Whatever runs the job, the feature must be defined once — a name, an entity key, an aggregation over a window — and materialized to two places: the online store for inference and the offline store for training.

-- Feature: signups from the same device fingerprint in the trailing 5 minutes.
-- Entity key: device_id. Event time from the producer, watermark tolerates 30s of lateness.
CREATE TABLE signup_events (
  device_id      STRING,
  account_id     STRING,
  ip_prefix      STRING,
  event_ts       TIMESTAMP(3),
  WATERMARK FOR event_ts AS event_ts - INTERVAL '30' SECOND
) WITH ('connector' = 'kafka', 'topic' = 'marketplace.signups', ...);

INSERT INTO online_features_signup_velocity   -- Redis sink, keyed by device_id, TTL 15m
SELECT
  device_id,
  COUNT(*)                        AS signups_5m,
  COUNT(DISTINCT ip_prefix)       AS distinct_ip_prefixes_5m,
  HOP_END(event_ts, INTERVAL '30' SECOND, INTERVAL '5' MINUTE) AS feature_ts
FROM signup_events
GROUP BY device_id,
         HOP(event_ts, INTERVAL '30' SECOND, INTERVAL '5' MINUTE);  -- sliding: 5m window, 30s slide

Windows, watermarks & late data

A streaming feature is an aggregate over a window, and every correctness question reduces to: which events are in the window, and when is the window's value final? Three window shapes cover the feature catalogue — tumbling (fixed, non-overlapping: bids per market per minute), sliding/hopping (overlapping: signups in the trailing 5 minutes, updated every 30 s), and session (gap-based: an account's actions until 20 minutes of silence). Sliding windows are the common choice for velocity features and the expensive one — each event lands in several windows, so state multiplies by window ÷ slide.

Event time versus processing time is the trap. The fraud feature must count signups by when they happened, not when the message reached the job — otherwise a Kafka partition that lags by two minutes makes a burst look spread out. Event-time processing needs a watermark: the job's declaration that "I have now seen everything up to time T, give or take my tolerance". Events arriving after the watermark passed their window are late, and the design decision is what to do with them: drop (simplest; fine for velocity counters where a 30-second-late event barely matters), update the emitted value (correct; requires the sink to accept upserts and consumers to tolerate change), or side-output to a reconciliation table (auditable; the batch path picks them up).

The online-store write path

Where the streamed value lands is half the design. The batch path materializes latest values into the online store on a schedule; the streaming job writes directly, continuously. The two paths meet in the same keyspace, which creates three rules:

• Key by entity, version by time. Every write carries feature_ts. A slower batch materialization must not overwrite a fresher streamed value; compare timestamps on write (Redis Lua, DynamoDB condition expressions) or namespace the two paths and resolve at read.
• TTL streamed features. A 5-minute velocity feature for a device that never signs up again should not live in Redis forever. Set TTLs to the window length plus margin; a missing key reads as zero, which is correct.
• Dual-write to the offline store. The same job that writes Redis must append the emitted values (entity, feature_ts, value) to the lakehouse — Iceberg via Flink's sink, Delta via Structured Streaming — because that history is the only way to build a leakage-free training set for a streamed feature. Skip this and the feature is untrainable.

  Kafka: marketplace.signups ──▶ Flink job: signup_velocity_5m
                                    │ keyed state (device_id), hop window, watermark 30s
                                    ├──▶ Redis  (online)   key device:{id}:signup_velocity_5m  TTL 15m
                                    │                      value {signups_5m, distinct_ips, feature_ts}
                                    └──▶ Iceberg (offline) features.signup_velocity_5m
                                                           (device_id, feature_ts, signups_5m, …)  append-only
                                                           ▲
  Nightly batch (dbt/Spark) ───────────────────────────────┘ reconciliation: recompute the same window
                                                             from raw events, diff vs streamed values,
                                                             alert if p99 |Δ| > tolerance

Stream–batch consistency: the second-definition problem

Here is the failure that actually ships. The training set for the fraud model is built in SQL over raw signup events, in batch, with a clean 5-minute window. The serving feature is computed in Flink, with a 30-second watermark, a hop of 30 seconds, and late events dropped. Same name, two definitions. Offline the feature is slightly "too complete" (it saw the late events); online it is slightly sparse. The model learns thresholds on the complete version and applies them to the sparse one. AUC drops from 0.94 offline to 0.85 in production, and nobody can find a bug because there isn't one — there are two correct implementations of different things.

Four ways out, in rising order of rigor:

1. Log the served value. Have the inference service record the exact feature vector it scored; build the training set from those logs rather than recomputing. Cheapest fix, trains only on what the model would have seen, but you cannot backfill a new feature.
2. One code path, two triggers. Structured Streaming lets the same query run as a micro-batch job and as a nightly batch over the lakehouse. Latency floors at the trigger, but the definition is shared.
3. Replay the streaming job over history. Run the Flink job against the Kafka retention (or a lakehouse-backed replay) to produce the offline history with the exact same windowing and watermark. Needs long retention or Tiered Storage, and takes as long as the stream is long unless the job is parallelized.
4. Reconcile continuously. Keep both paths but recompute the batch version of every streamed feature nightly, diff it against the streamed history, and alert on drift beyond a tolerance. This does not prevent the skew; it makes it visible within a day instead of at the next model review.

Worked example: fraud velocity at GridDP

GridDP's fraud model scores every signup and every new listing. Its batch features (payment-method age, KYC outcome, chargeback history) come from the nightly materialization. Its two streamed features — signup_velocity_5m keyed by device fingerprint and listing_velocity_15m keyed by host — run in one Flink job reading the marketplace event topics with a 30-second watermark. The job writes Redis with a 15-minute TTL and appends to Iceberg. The nightly reconciliation recomputes both windows from the raw topic mirrored in the lake and diffs; tolerance is 2% at p99, and the one time it fired it caught a producer that had switched from event time to send time after a deploy.

Cost shape: the Flink job is two task managers and about 4 GB of keyed state; the Redis keyspace stays under a million entries because of the TTL; the Iceberg append is ~2 GB/day before compaction. The point is not that this is small — it is that it is small because only two features stream. The first design had eleven.

When to choose what

Next, the features that are not numbers at all: 07 · Embeddings, Vector Data & RAG Pipelines.

✓ Check yourself

Answer from memory before opening each one.

1. A ranking model scores offers in 50 ms. A colleague says it therefore needs a streaming pipeline for host_reliability_7d. What question do you ask, and what does the feature need instead?

   Show answer

   Ask what it costs the model to see a value that is an hour stale. For a seven-day aggregate the answer is nothing, so the feature needs fast reads from the online store, not fresh computation. Stream only when the value changes inside the batch interval and the decision depends on it. See which features need to stream.

2. A feature is advertised as sub-second, but it sits behind a five-minute tumbling window. What is its real freshness, and which three components make up end-to-end freshness?

   Show answer

   It is a five-minute feature. Freshness runs from the event to the value being visible to the model, and it adds transport lag to the broker, compute lag including window close and processing, and write lag to the online store. See freshness SLAs in numbers.

3. A Kafka partition lags by two minutes. Why does a velocity feature counted on processing time look wrong, and what does the job need to count on event time correctly?

   Show answer

   A burst arrives late and spread out, so processing-time counts make it look slower than it was. Count on event time and set a watermark, the job's statement that it has seen everything up to time T within a tolerance, then decide what to do with later events. See windows, watermarks and late data.

4. You are asked to raise the watermark tolerance from 30 seconds to 5 minutes to catch more late events. What does it cost, and where do you take the number from?

   Show answer

   Every value is emitted about five minutes later, so the feature gets staler while it gets more complete. Choose it from the measured lateness of the producer, such as the p99 of arrival time minus event time, and record it with the definition so training uses the same rule. See windows, watermarks and late data.

5. A nightly batch materialisation writes a feature to Redis while the streaming job also writes it. Name two rules for that shared keyspace and the failure each prevents.

   Show answer

   Version each write with feature_ts and reject older ones, so a slow batch run cannot overwrite a fresher streamed value. Set a TTL of the window length plus margin, so keys for entities that went quiet expire and read as zero. Also append every emitted value to the offline store, or the feature has no history to train on. See the online-store write path.

6. Training reads a clean five-minute window computed in batch, and serving reads Flink output that dropped late events. Offline AUC is fine and live precision falls. What is the failure, and how do you guarantee a match with a number?

   Show answer

   Two definitions share one name: offline counts late events and online does not, so thresholds learned on one are applied to the other. Log served feature vectors for training, recompute the window from raw events each night, and page when the p99 difference exceeds a stated tolerance. A mechanism without a tolerance is a hope. See stream-batch consistency.

Takeaway

• Freshness is a property of each feature, so batch-compute and materialise by default and stream only what changes inside the batch interval and drives a decision.
• Write the freshness SLA in seconds before choosing the engine, and measure it end to end from event to visible value.
• Flink fits seconds and keyed state, Spark Structured Streaming fits minutes and shared batch code, and managed offerings change the operations, not the semantics.
• Count on event time with a watermark chosen from measured lateness and recorded with the feature definition.
• A streamed feature needs versioned writes, a TTL and an append to the offline store, or it cannot be trained on.
• The main risk is a second definition of the same feature, so log served vectors, reconcile nightly against raw events and state the tolerance.

Some features are not numbers, and their pipelines have their own failure modes. → Embeddings, Vector Data & RAG Pipelines

Further reading

• Timely Stream Processing — Apache Flink documentation, accessed 2026. Event time, processing time and watermarks as Flink defines them.
• Structured Streaming Programming Guide — Apache Spark documentation, accessed 2026. Micro-batch triggers, event-time windows and watermarks for late data.
• The Dataflow Model — Akidau et al., VLDB 2015. The model behind windows, triggers and late data in unbounded streams.
• Rules of Machine Learning: Best Practices for ML Engineering — Google, accessed 2026. Why logging features at serving time avoids training and serving skew.