Interview Prep Hub

Production Data Pipelines

Prep for the 60-minute, discussion-based production-pipeline interview — a deep, whiteboard-friendly conversation about building robust, real-world data systems. It centers on three levers the interviewer will pull: pipeline design & orchestration, handling late-arriving data, and data reconciliation & validation. This guide is built as a progression — every topic climbs from foundational to staff-level, so you can meet the question wherever it lands and then visibly level your answer up. Tool-agnostic first (patterns over products), with dbt and Airflow as the worked examples and the major alternatives named throughout.

How to read this guide — the skill ladder

A discussion interview isn't graded on whether you know one right answer; it's graded on how far up the ladder you can climb as the conversation deepens. Every content chapter is written against these five levels, and shows you how to move between them out loud.

Level	What it demonstrates
L1 · Foundational	Knows the vocabulary and building blocks; can describe a simple, correct batch pipeline.
L2 · Practitioner	Builds idempotent, retryable pipelines; handles common failure modes; uses orchestration correctly.
L3 · Senior	Designs for late-arriving data, incremental processing, and reconciliation; reasons explicitly about trade-offs; owns reliability and SLAs.
L4 · Staff	Architects across teams and at scale; data contracts, cost, streaming/lakehouse, failure-domain thinking, schema and architecture evolution.
L5 · Principal	Org-level platform strategy, standards, and migration. The aspirational ceiling; touched lightly.

Capstone: The Design Session Drive the 60-min whiteboard, worked end-to-end Drill: Practice Questions ~30 Q&A · hide-show answers Execution: Day-Of Tactics Framework, whiteboard moves, red flags

Study pathsPick the plan that fits the time you have

a week

Day 1: 00, 01 (orient + the leveling model) → 02 (foundations).
Day 2: 03 (orchestration).
Day 3: 04 (late-arriving data — a named topic, go deep).
Day 4: 05 (incremental) + 06 (reconciliation — a named topic).
Day 5: 07 (reliability) + 08 (scale).
Day 6: 09 (the design session — do the worked scenario on a whiteboard yourself).
Day 7: Drill 10. Reread 11. Sleep.

two days

00, 01, 02, then the three named-topic chapters — 04 (late data), 06 (reconciliation), plus 03 (orchestration) — then 09 (design session), drill 10, and 11. Skim 05, 07, 08.

an evening

01 (how the interview is graded + the ladder), the idempotency section of 02, 04 (late data), 06 (reconciliation), and 09 (the framework). Those are the load-bearing pieces of the whole loop.

Section A · Orient

00 Start Here What this interview is, the skill-ladder model that runs through the whole guide, the reading order, and a study schedule. Read first. 01 The Discussion Interview & the Leveling Model How a 60-minute discussion is actually graded, how to structure an answer, how to signal progression from foundational to staff, and how to whiteboard so the interviewer follows your thinking.

Section B · Design & Orchestration

02 Pipeline Design Foundations The anatomy of a pipeline, batch vs streaming, ETL vs ELT, the DAG mental model, medallion modeling, and the one property everything rests on — idempotency. L1 → L2 03 Orchestration What an orchestrator really gives you — scheduling, dependencies, retries, backfills, sensors, SLAs. Airflow deep, plus Dagster / Prefect / Temporal, asset-vs-task orchestration, and the antipatterns. L2 → L4

Section C · The Hard Parts

04 Late-Arriving Data Event vs processing time, watermarks and allowed lateness, windowing, lookback/reprocessing, late-arriving facts and dimensions, SCD2, bitemporal modeling, and lambda vs kappa. One of the three named topics. CriticalL2 → L4 05 Incremental Processing & Backfills Append vs merge vs insert-overwrite vs microbatch, watermark bookkeeping and boundary bugs, CDC, snapshots, and designing backfills that don't double-count. L2 → L3 06 Reconciliation & Validation Source-to-target reconciliation and control totals, the data-quality dimensions, tests and contracts, Write-Audit-Publish, circuit breakers, quarantine, and anomaly detection. One of the three named topics. CriticalL2 → L4

Section D · Production-Grade

07 Reliability & Observability Delivery semantics and exactly-once, the outbox pattern, retries and dead-letter queues, lineage, freshness SLAs/SLOs, alerting, and incident response for data. L3 → L4 08 Scale & Architecture Partitioning and file formats, the lakehouse table formats (Iceberg / Delta / Hudi), streaming with Kafka + Flink/Spark, the batch-vs-streaming decision, cost engineering, and migration. L4 → L5

Section E · Putting It Together

09 The Design Session The capstone. A repeatable framework for driving the 60-minute whiteboard — clarify, SLAs, architecture, deep dives, trade-offs, evolution — walked end-to-end on a full worked scenario, showing how to level up live. 10 Practice Questions ~30 Q&A across design, orchestration, late data, incremental, reconciliation, reliability, and scale — each noting how a staff answer goes beyond a senior one. Drill mode hides the answers. Drill mode 11 Day-Of Tactics The answer framework, whiteboarding moves, how to demonstrate progression, questions to ask, red flags, and recovery patterns. Reread the morning of.