Drill: Real-Time Features for Fraud & Ranking

The prompt

Two requirements in one sentence: freshness in seconds, and offline/online consistency. Both must appear on your board.

Clarifying questions

• Latency budget for the score? "150 ms end to end for the authorisation; the model gets 50, feature retrieval must fit in ~10." → In-memory store, multi-get, no stream query at request time.
• Which features need to be fresh? "Velocity features — transactions and amount in the last 1, 5, 60 minutes per card, per device, per merchant; new-device flags; distance from last transaction." → A small set of windowed aggregates over a few entities, plus request-time computations.
• Volume? "~3,000 transactions/s peak; ~8 events per transaction across auth, device, and session." → ~25k events/s into the stream jobs.
• Entities? "200M cards, 50M devices, 5M merchants." → Online store sized by hot entities, not all — most cards are idle; TTL and lazy population matter.
• Labels? "Chargebacks, T+30–90 days; confirmed fraud from the ops team sooner." → Training uses matured labels; evaluation lag is months; velocity features need logged values for training.
• What exists? "Kafka, Flink for another team, a warehouse, Redis somewhere." → Build on it.

Move 1 · Requirements & estimate

Three classes of feature

Before drawing, sort the features. Each class has its own path, and mixing them up is where designs get expensive.

• Batch features (most of them). Card age, historical chargeback rate, merchant category risk, 90-day spend profile. Computed nightly in the warehouse, materialised to the online store. Staleness of a day is fine.
• Streaming features (the velocity set). Count and sum over the last 1/5/60 minutes per card, device, merchant; time since last transaction; distinct merchants in the last hour. Computed by a stream job holding per-key windowed state, written to the online store on every update.
• Request-time features (computed in the scoring call). Amount vs the card's average (needs the average from the store), distance between this transaction's location and the last one (needs the last location from the store), "is this device new for this card" (a membership check). Cheap, derived from retrieved values plus the request itself. They are never stored — they are logged.

The board

   EVENTS (Kafka, keyed)              STREAM FEATURES                     ONLINE STORE              SCORING (150 ms)
  ┌────────────────┐               ┌──────────────────────┐            ┌─────────────────┐       ┌──────────────────┐
  │ auth_events    │──▶ by card ──▶│ Flink job            │ upsert ──▶ │ Redis cluster   │◀──────│ fraud scorer     │
  │ device_events  │──▶ by device─▶│  keyed state         │            │ card:{id}   TTL │ mget  │  1 assemble vec  │
  │ session_events │──▶ by merch ─▶│  windows 1m/5m/60m   │            │ device:{id} TTL │ 10ms  │  2 request-time  │
  └───────┬────────┘               │  event-time+watermark│            │ merch:{id}  TTL │       │  3 model 50ms    │
          │                        │  checkpoints (RocksDB)│           └────────▲────────┘       └────────┬─────────┘
          │                        └──────────┬───────────┘                     │ nightly materialise    │ feature log
          │ same events                       │ same definitions                │                        │ (txn_id, vector,
          ▼                                   ▼                                 │                        │  feature ts)
  ┌────────────────┐               ┌──────────────────────┐            ┌─────────────────┐                ▼
  │ raw lake       │──────────────▶│ batch features       │───────────▶│ offline store   │◀──────── logged features
  │ (Iceberg)      │               │ (warehouse, nightly) │            │ (history)       │          + labels (T+30–90d)
  └────────────────┘               └──────────────────────┘            └────────┬────────┘
                                                                                ▼
                       RECONCILE nightly: recompute velocity features from raw as-of txn ts  vs  logged ── alert on Δ
  ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   OPERATE: consumer lag SLO ◆  watermark lateness ◆  store hit-rate ◆  degradation mode ◆  cost (Redis GB, Flink slots)

Event time, windows & lateness

The streaming features are windowed aggregates, and the correctness of a windowed aggregate depends on which clock you use. Say "event time" before the interviewer does.

• Event time, not processing time. "Transactions in the last 5 minutes" means 5 minutes of the card's timeline, not 5 minutes of wall clock at the Flink task. Events are stamped at the source; the job orders and windows by that stamp.
• Watermarks and lateness. Events arrive out of order (mobile clients, retries). A watermark says "I've seen everything up to time w, probably." Choose the allowed lateness from the data: if 99.9% of events arrive within 3 s, a 5 s watermark delay trades 5 s of freshness for near-complete windows. Late events beyond that are applied as corrections or counted in a side output — say which, and why (fraud: apply corrections; the cost of a missed velocity spike is high).
• Window type. Sliding windows with a 1-second slide are expensive at 25k events/s; use continuous aggregation over keyed state — keep a ring buffer or a small sorted set of recent timestamps per key and compute the 1/5/60-minute counts on each event. State per key is tiny; the 60-minute window bounds it.
• State and checkpoints. Keyed state in RocksDB, checkpointed to object storage every 30–60 s; on failure the job restores and replays from Kafka offsets. Writes to Redis are idempotent (last-write-wins with the event timestamp as a version), so replay is safe.

Depth: Deep dive 06 · Streaming Features & Real-Time Inference Data.

Keeping stream and batch consistent

The second half of the prompt. There are three mechanisms; a senior answer uses all three and knows which one is the safety net.

1. One definition, two engines. The feature "card_txn_count_5m" is defined once (in the registry, as a declarative spec or a shared library) and compiled or reused in both the Flink job and the batch backfill. Where engines differ (Flink SQL vs warehouse SQL), keep the definitions in the narrowest shared dialect and test them with the same fixtures.
2. Log what was served. The scorer logs the full feature vector with the transaction id and the feature timestamps. The training set is built from the log joined to matured labels — so the model trains on exactly what production saw, including its lateness and its bugs. This is the strongest guarantee available and it costs one Kafka topic.
3. Reconcile. Nightly, recompute the velocity features from raw events as-of each logged transaction's timestamp and diff against the logged values. Report the distribution of deltas per feature; alert when the p95 delta crosses a threshold. This is how you catch a watermark that is too aggressive, a UDF that diverged, or a Redis TTL eating a feature.

Failure modes & degradation

• Flink falls behind (consumer lag). Features go stale silently. Expose lag as a metric; the scorer reads the feature timestamp from the store and, if it is older than the SLA, treats the velocity features as missing and lets the model's missing-value handling take over (train with that dropout so it's a known state). Page at lag > 30 s.
• Redis unavailable. Score with batch features from a local fallback cache or defaults; flag the decision as degraded; lower the auto-approve threshold if policy allows. Never fail the authorisation on a feature-store outage.
• Hot keys. A merchant doing 500 txn/s concentrates on one Flink key. Pre-aggregate per merchant per second at the source, or split the key with a salt and merge on read.
• Backfilling a new streaming feature. Replay Kafka (if retention allows) or compute from the raw lake with the same definition; write history to the offline store; only then switch the live job on. The online store starts empty for the feature and warms as events arrive — say the warm-up window.
• Schema change on the event. Schema registry with compatibility checks; the Flink job pins a schema version; a breaking change is a new topic, not a surprise.

Trade-offs

What the interviewer is listening for

• The three feature classes, and that only the velocity set goes on the stream.
• "Event time" and "watermark" unprompted, with a lateness number derived from the data.
• A latency budget broken into slices and an online store sized by the hot set, not the universe.
• Feature logging as the training path; reconciliation as the alarm.
• Degradation modes that keep the authorisation flowing.
• Idempotent writes and replay from offsets — the reason failures are boring.

Follow-up probes

• "Ranking wants the same thing: the last five clicks. Same design?" — Same spine, different tolerances: click features tolerate loss (drop late events, shorter TTLs), the store holds a small list per user rather than counts, and the budget is shared with retrieval of candidates. Reuse the Flink pattern and the registry; don't reuse the fraud job.
• "What's your freshness SLO and how do you measure it?" — End-to-end: event source timestamp → visible in Redis, sampled by writing a canary event per second per partition and reading it back. Target p99 ≤ 5 s; alert at 15 s.
• "Exactly-once?" — Kafka → Flink is exactly-once with checkpoints; Flink → Redis is at-least-once made idempotent by versioned upserts. Say that you don't chase distributed exactly-once into the store.
• "Cost?" — Flink: ~15 slots ≈ a handful of nodes; Redis: 30 GB ×3 replicas; the feature log topic at 3,000 msg/s × 2 KB ≈ 500 GB/day raw → compact to the lake nightly. Name the log as the surprising line item.

The answer at four levels

Next, a different kind of feature — vectors, corpora and the pipelines that keep them fresh: 06 · Drill: Embedding & RAG Data Infrastructure.

✓ Check yourself

Answer from memory before opening each one.

1. The model needs "this transaction's amount divided by the card's 90-day average amount". Which class of feature is it, what is stored, and what is logged?

   Show answer

   It is a request-time feature. The 90-day average comes from the store as a batch feature, and the division happens in the scoring call with the amount from the request. The ratio is never stored, but it is logged with the rest of the vector. See three classes of feature.

2. A "transactions in the last 5 minutes" feature is windowed on event time with a 5 s watermark delay. A mobile client delivers an event 20 s late. What happens, and what did the 5 s buy you?

   Show answer

   The event falls behind the watermark, so it is handled by the lateness policy: for fraud, apply it as a correction rather than drop it. The 5 s delay traded 5 s of freshness for near-complete windows when most events arrive within about 3 s. See event time, windows and lateness.

3. Why train on the logged feature vectors and not recompute streaming features offline, and what is the recompute still used for?

   Show answer

   Recomputation "should" match, but that word hides every skew bug, while the logged vector is exactly what production saw, bugs included. Keep recomputation for backfills and for the nightly reconciliation that checks the log. See keeping stream and batch consistent.

4. The Flink job falls two minutes behind. What does the scorer see, what should it do, and what alert fires?

   Show answer

   The store still returns values, only stale ones, so the failure is silent. The scorer reads the feature timestamp, treats velocity features older than the SLA as missing, and relies on the model having been trained with that missing state. Page on consumer lag over 30 s. See failure modes and degradation.

5. Flink restarts from a checkpoint and replays 40 seconds of Kafka. Why does Redis not end up with wrong or double-counted values?

   Show answer

   Flink keeps its own state exactly-once through checkpoints, and the writes to Redis are versioned upserts using the event timestamp, so a replayed write has no further effect. The design makes the store idempotent rather than chasing distributed exactly-once. See event time, windows and lateness.

6. The home-page team asks to reuse the fraud Flink job for "the last five clicks". What do you reuse and what do you change?

   Show answer

   Reuse the pattern, the registry and the idempotent write approach, but not the fraud job itself. Click features tolerate loss, so late events can be dropped, TTLs are shorter, the store holds a small list per user rather than counts, and retrieval shares its budget with candidate retrieval. See the follow-up probes.

Takeaway

• Sort features into batch, streaming and request-time, and stream only the small velocity set that needs seconds-level freshness.
• Window by event time with a watermark delay chosen from the observed lateness. For fraud, apply late events as corrections because a missed spike is costly.
• Aggregate continuously in keyed state with checkpoints, and write to the store with versioned upserts so a replay is safe.
• Train on the logged feature vectors. Use offline recomputation for backfills and for the reconciliation check, not as the training path.
• Size the online store by the hot set of active entities, and split the latency budget into slices for retrieval and the model.
• Degrade, do not fail: stale or missing velocity features lead to defaults and a degraded flag, never a failed authorisation.

Those were numeric and keyed features. The next drill is vectors, corpora and the pipelines that keep them fresh. → Drill: Embedding & RAG Data Infrastructure

Further reading

• Timely Stream Processing — Apache Flink documentation. Event time, processing time and the watermark idea.
• Generating Watermarks — Apache Flink documentation. Bounded out-of-orderness and how lateness is configured.
• Checkpoints — Apache Flink documentation. How keyed state is snapshotted and restored after failure.
• State Backends — Apache Flink documentation. RocksDB versus heap state, the choice behind per-key windowed state.