Batch vs Streaming

Batch, micro-batch, streaming

Before you can defend a choice, define the options crisply. The distinction is not "fast vs slow" — it's what triggers computation and what the unit of work is.

Micro-batch is the pragmatic middle: Spark Structured Streaming's default execution model chops an unbounded stream into a series of tiny batch jobs. It buys most of the freshness of streaming with the operational familiarity and throughput of batch, which is why it's the workhorse for a huge share of "near-real-time" pipelines. True record-at-a-time streaming (Flink) is what you reach for when single-digit-second or sub-second latency and rich per-key state actually matter.

What actually drives the choice

The decision is not about technology preference. It's driven, in order, by:

1. Freshness SLA. The single dominant input. If a consumer needs data within seconds (fraud scoring, live ops dashboards, alerting), streaming earns its keep. If "by 6am" or "within the hour" is fine (finance reporting, most BI, ML training sets), batch or micro-batch wins comfortably. Ask what the consumer actually does with fresher data — often the answer reveals the real SLA is looser than the ask.
2. Cost. Streaming means always-on compute and standing infrastructure (a Kafka/Flink cluster running 24/7) versus batch compute you spin up, run, and tear down. At modest volume a nightly warehouse job can be an order of magnitude cheaper than an equivalent streaming stack.
3. Complexity & operational burden. Streaming pulls in watermarks, windowing, state stores, checkpointing, backpressure, and 24/7 on-call. Batch failures are re-runnable in the morning; a stalled stream is a page.
4. Correctness under late/out-of-order data. Batch sees a whole bounded window at once, so late data is "just there." Streaming must decide when a window is done in the face of events that haven't arrived yet — a genuinely harder correctness problem (below).
5. Team skill. A team fluent in SQL and dbt will ship a reliable batch platform far faster than they'll operate Flink well. Match the architecture to who has to run it at 3am.

Time, windows & correctness

The hardest part of streaming is that events arrive late and out of order, so you must distinguish two clocks:

• Event time — when the event actually happened (stamped at the source). What you almost always want to aggregate by.
• Processing time — when your engine saw it. Easy, but wrong for anything time-windowed, because a mobile client that was offline for an hour will inject "old" events into "now."

A watermark is the engine's assertion that "I don't expect any more events older than time T," which is what lets a windowed aggregation close and emit. Set it too tight and you drop late events; too loose and you hold state and delay output. This is the crux of streaming correctness — and it has a lot of depth.

The architectural takeaway: batch's correctness is easy because it operates on a closed window; streaming trades that ease for freshness, and pays in watermark tuning and late-data handling.

Exactly-once & reprocessing

"Exactly-once" is a property of the whole path, not a checkbox. In practice you get it by combining at-least-once delivery with idempotent or transactional sinks, plus checkpointed operator state.

• Flink achieves end-to-end exactly-once via distributed checkpointing (Chandy–Lamport barriers) plus a two-phase-commit sink connector — output is committed only when a checkpoint completes.
• Spark Structured Streaming gives exactly-once with replayable sources (Kafka offsets), a write-ahead log, and idempotent sinks (e.g. a Delta table with transactional commits).
• Kafka offers transactions and idempotent producers, giving exactly-once within the Kafka-to-Kafka boundary; the sink still has to hold up its end.

Reprocessing is where the two paradigms diverge most sharply, and it's a favorite deep-dive:

• Batch reprocessing is trivial — a bug in yesterday's logic means re-running yesterday's partition. Idempotent overwrite of a partition is the whole story.
• Streaming reprocessing is a design problem. To fix a bug you must replay history from a retained log (Kafka retention or a lake replay), rebuild state, and reconcile with already-emitted results. This is precisely the pain Kappa architecture tries to make routine and Lambda tries to sidestep.

Lambda vs Kappa

Two reference architectures for combining freshness with correctness:

• Lambda runs two paths: a speed layer (streaming, approximate, low-latency) and a batch layer (slow, complete, authoritative) that periodically recomputes and corrects the speed layer's output. The serving layer merges them. Cost: you maintain the same business logic twice, in two codebases, and reconcile them.
• Kappa keeps one path — streaming only. Reprocessing is done by replaying the log through the same code with new logic. Simpler conceptually and no duplicated logic, but it demands durable, replayable logs and an engine that can reprocess history fast.

The engine landscape

Match the engine to the mode and to the correctness/latency need. You should be able to place each of these on the board and say why.

A short Structured Streaming reader shows how thin the code difference between batch and micro-batch actually is — the same DataFrame API, a different source and a trigger:

# Micro-batch stream: read Kafka, aggregate by event-time window,
# tolerate 10 minutes of lateness, write exactly-once to Delta.
from pyspark.sql import functions as F

events = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "orders")
    .load())

agg = (events
    .select(F.from_json(F.col("value").cast("string"), schema).alias("e"))
    .select("e.*")
    .withWatermark("event_ts", "10 minutes")          # late-data tolerance
    .groupBy(F.window("event_ts", "5 minutes"), "region")
    .agg(F.sum("amount").alias("revenue")))

(agg.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "/chk/orders")       # state + exactly-once
    .trigger(processingTime="30 seconds")              # micro-batch cadence
    .start("/lake/gold/revenue_5m"))

The decision matrix

This is the artifact to reproduce on the whiteboard. Score the requirement against each axis; the column with the better fit for your dominant requirement wins that path.

Notice batch wins six of seven axes; streaming wins the one that, when it matters, matters absolutely — latency. That asymmetry is the whole trade-off in a sentence.

When to choose what

Deliver these as defended defaults, then name the condition that flips them.

Storage sits underneath all of this — where the batch and streaming outputs land, and in what format, is the next fork. Continue to 06 · Lakes, Warehouses & Lakehouses.

✓ Check yourself

Answer from memory before opening each one.

1. A mobile client was offline for an hour and then uploads its events. If you window by processing time, where do they land, and what does a watermark set too tight or too loose cost you?

   Show answer

   They land in the current window, not the hour in which they happened, so the aggregate is wrong. Window by event time. A watermark that is too tight drops late events. One that is too loose holds state longer and delays output. See time, windows and correctness.

2. A stakeholder asks for real-time revenue. The only reader is a finance dashboard opened at 9 a.m. What do you ask, and what is the likely outcome?

   Show answer

   Ask what action is taken on the data and how stale it can be before the action is wrong. For a dashboard read at 9 a.m. the answer is hours, so a nightly or hourly batch meets it for far less cost and on-call load. See what drives the choice.

3. A transform bug is found, and yesterday's numbers are wrong. Compare the fix in a batch pipeline and in a streaming pipeline.

   Show answer

   In batch you re-run yesterday's partition with an idempotent overwrite. In streaming you must replay history from a retained log, rebuild operator state, and reconcile with results already emitted, so retention has to cover the repair window. See exactly-once and reprocessing.

4. A Flink job calls a payment provider for each record and is configured for exactly-once. After a restart some customers get two emails. Why?

   Show answer

   Exactly-once covers state and the sink commit, not side effects. A replay after a checkpoint re-runs the call to the provider. Send with an idempotency key, or move the effect behind a transactional or idempotent sink. See exactly-once and reprocessing.

5. A teammate proposes Lambda: a streaming layer for speed and a batch layer to correct it, so the numbers are right in the end. What cost do you raise, and what do you recommend instead?

   Show answer

   The same business logic lives in two codebases that must agree, and the serving layer has to merge them. Prefer one path, either a streaming-first design that replays the log to reprocess, or incremental batch if the SLA allows. See Lambda vs Kappa.

6. A platform has fraud blocking in seconds, daily finance reports, and weekly model training. Where does streaming go?

   Show answer

   Only on the fraud path, which acts on data within seconds and loses value as it ages. Batch the rest, since nothing consumes it faster than daily. Splitting by SLA keeps the standing cluster and pager to one path. See when to choose what.

Takeaway

• Batch, micro-batch and streaming are points on a latency and cost curve. Most platforms run more than one, chosen per data path.
• The freshness SLA decides first. Ask what action is taken on the data and how stale it can get before the action is wrong.
• Streaming costs always-on compute, state, watermarks and on-call. Batch wins on cost, complexity, late data and reprocessing, and streaming wins only on latency.
• Aggregate by event time. A watermark trades completeness against latency, and batch avoids that choice because its windows are closed.
• Exactly-once is a property of the whole path: checkpointed state plus a transactional or idempotent sink. It does not cover external side effects.
• Reprocessing is a re-run in batch and a replay with state rebuild in streaming.
• Lambda duplicates logic. Default to one path, unified or incremental, unless a requirement forces two.

Whichever mode you pick, the output has to land somewhere, in some format. → Lakes, Warehouses & Lakehouses

Further reading

• The Dataflow Model — Akidau et al., VLDB, 2015. Event time, windows, watermarks and triggers as one model for bounded and unbounded data.
• Stateful Stream Processing — Apache Flink documentation. Checkpointing and how state is recovered after a failure.
• Lightweight Asynchronous Snapshots for Distributed Dataflows — Carbone et al., 2015. The barrier-based snapshot algorithm behind Flink checkpoints.
• Structured Streaming Programming Guide — Apache Spark documentation. The micro-batch model, watermarks, triggers and output modes.
• Kafka message delivery semantics — Apache Kafka documentation. Idempotent producers and transactions, and what exactly-once covers.