Start Here
A study guide for the 60-minute, discussion-based production data pipeline interview — built as a progression, so you can meet any question at its level and then climb.
The interview in one paragraph
This is a 60-minute, discussion-based session. You and the interviewer pick a production data-pipeline scenario and go deep — no LeetCode, no take-home, just a conversation about how you'd build a robust, real-world data system, with a whiteboard available if a sketch helps. The stated territory is pipeline design and orchestration, strategies for handling late-arriving data, and approaches to data reconciliation and validation. The company runs dbt and Airflow, but the interview is about your judgment, not their toolchain — so this guide teaches the patterns first and treats specific tools as worked examples.
A discussion interview measures depth and adaptability. The interviewer starts somewhere reasonable and keeps asking "and then what?" — pushing until you reach the edge of your experience. You win not by reciting one correct architecture, but by showing you can start simple and correct, then layer in production concerns on your own initiative: idempotency, late data, reconciliation, failure modes, cost. That upward movement is the whole game.
The idea that runs through everything
Because depth is what's graded, this guide is organized around a skill ladder. Every content chapter is written to climb it, and — critically — teaches you how to move between rungs out loud in the room.
| Level | What it demonstrates |
|---|---|
| L1 · Foundational | Knows the vocabulary and building blocks; can describe a simple, correct batch pipeline. |
| L2 · Practitioner | Builds idempotent, retryable pipelines; handles common failure modes; uses orchestration correctly. |
| L3 · Senior | Designs for late-arriving data, incremental processing, and reconciliation; reasons explicitly about trade-offs; owns reliability and SLAs. |
| L4 · Staff | Architects across teams and at scale; data contracts, cost, streaming/lakehouse, failure-domain thinking, schema and architecture evolution. |
| L5 · Principal | Org-level platform strategy, standards, and migration — the aspirational ceiling. |
You don't need to live at L5. You need to enter a topic at L1–L2 confidently and climb two rungs under questioning. A candidate who says "the naive version is a nightly full refresh; in production I'd move to an idempotent incremental merge with a lookback window because late events would otherwise be silently dropped" has just walked L1 → L3 in one sentence. That is the move this guide drills into you. Chapter 01 teaches the mechanics of doing it.
The three levers
The interviewer has three explicit topics to pull on. Expect to spend real time on each; the guide gives each its own deep chapter.
- Pipeline design & orchestration — how you'd structure ingestion, storage, transformation, and scheduling; dependencies, retries, backfills. (Chapters 02, 03.)
- Handling late-arriving data — the classic that separates people who've run pipelines from people who've only built them. Event vs processing time, watermarks, lookback windows, late facts and dimensions. (Chapter 04, with 05.)
- Reconciliation & validation — how you prove the data is right: source-to-target reconciliation, control totals, tests, contracts, and stopping bad data before it ships. (Chapter 06.)
Everything else — reliability, observability, and scale (07, 08) — is the production context that lets you answer those three like someone who's carried a pager.
The guide in reading order
Five sections. The numbering is the order to read in.
Section A — Orient
| Chapter | Why |
|---|---|
| 01 · The Discussion Interview | How the format is graded, how to structure an answer, how to signal progression, and how to whiteboard |
Section B — Design & Orchestration
| Chapter | Why |
|---|---|
| 02 · Pipeline Design Foundations | Anatomy, batch vs streaming, ETL vs ELT, the DAG model, and idempotency — the property everything rests on |
| 03 · Orchestration | Scheduling, dependencies, retries, backfills, sensors, SLAs — Airflow deep, plus Dagster/Prefect/Temporal and the antipatterns |
Section C — The Hard Parts
| Chapter | Why |
|---|---|
| 04 · Late-Arriving Data | Event vs processing time, watermarks, lookback, late facts/dimensions, SCD2, bitemporal — a named topic |
| 05 · Incremental Processing & Backfills | Merge vs insert-overwrite vs microbatch, watermarks, CDC, and backfills that don't double-count |
| 06 · Reconciliation & Validation | Recon & control totals, DQ dimensions, tests, contracts, Write-Audit-Publish, circuit breakers — a named topic |
Section D — Production-Grade
| Chapter | Why |
|---|---|
| 07 · Reliability & Observability | Delivery semantics, the outbox pattern, DLQs, lineage, freshness SLAs, and incident response |
| 08 · Scale & Architecture | Partitioning, file/table formats, the lakehouse, streaming, cost, and migration |
Section E — Putting It Together
| Chapter | Why |
|---|---|
| 09 · The Design Session | The capstone — a framework for driving the 60-min whiteboard, walked end-to-end on a real scenario |
| 10 · Practice Questions | ~30 Q&A with drill mode, each noting how the answer levels up |
| 11 · Day-Of Tactics | The framework, whiteboard moves, questions to ask, red flags. Reread morning of |
Study schedule
Three calibrations — the same paths are on the hub:
- A week: one section per day; on day 6 actually whiteboard the worked scenario in 09 out loud, on a timer.
- Two days: 00, 01, 02, the three lever chapters (03, 04, 06), 09, drill 10, 11. Skim 05, 07, 08.
- An evening: 01 (how it's graded + the ladder), the idempotency section of 02, all of 04 and 06, and the framework in 09.
On dbt, Airflow, and being tool-agnostic
The team uses dbt and Airflow, and you should be fluent enough to speak both — this guide uses them as the primary worked examples. But do not reduce your answers to "I'd add a dbt test" or "I'd write an Airflow sensor." Interviewers at this level are probing whether you understand the concept underneath the tool: why idempotency matters regardless of engine, what a watermark is whether or not your framework hands you one, what reconciliation proves that a schema test doesn't. Lead with the pattern, then name the tool that implements it — and mention the alternative (Dagster's asset model, Spark/Flink for streaming, Iceberg for table state, Great Expectations for validation) to show the concept is bigger than any one product.
When you can say "in Airflow I'd do this with a deferrable sensor; in Dagster this is just an asset dependency; the underlying idea is data-aware scheduling," you've signaled that you own the concept and merely use the tools. That single sentence often moves you a level.
What winning looks like
You don't need to have built every architecture in this guide. You need to be the candidate who:
- Clarifies requirements and SLAs before drawing boxes — volume, latency, freshness, correctness bar, consumers.
- Starts with a simple, correct, idempotent design and can defend why re-running it is safe.
- Proactively raises late-arriving data and shows a concrete strategy (lookback window + idempotent merge, watermarks, SCD2) rather than waiting to be asked.
- Treats reconciliation and validation as first-class — can describe how they'd prove the output matches the source and stop bad data from shipping.
- Names failure modes and how the pipeline degrades, retries, and recovers — like someone who's been on call.
- States trade-offs explicitly (latency vs cost vs correctness vs complexity) and levels the design up and down fluently.
If you can do those six things in a flowing conversation, you're in. Turn to 01 · The Discussion Interview.