Orchestration

What an orchestrator gives you

An orchestrator is not a compute engine and it's not a cron replacement — though people misuse it as both. It's the thing that runs your DAG (from Chapter 02) reliably. Concretely, it earns its keep by giving you:

• Scheduling — run this graph on a cadence, or when an upstream dataset updates.
• Dependency resolution — walk the DAG in topological order; run a task only once its upstreams succeed.
• Retries & backoff — automatically re-run a failed task N times with increasing delay, so transient failures self-heal (which only works if the task is idempotent).
• Backfill & catchup — run history: fill in the last 90 days of partitions on demand, or catch up runs missed while the scheduler was down.
• Sensors / triggers — wait for an external condition (a file lands, a partition is ready, an API is healthy) before proceeding.
• SLAs & observability — alert when a task misses its deadline; surface run history, logs, and durations.
• Parameterization — bind each run to its data interval so tasks are functions of a date, not of "now."

That last point is the throughline from Chapter 02: the orchestrator's most valuable habit is handing every task the data interval it's responsible for, which is exactly what makes idempotent, backfillable pipelines possible.

Airflow, worked

Airflow is the default in most shops (and the one this company runs alongside dbt), so it's worth being fluent. A DAG is a Python file that declares tasks and their dependencies; the scheduler executes them. The modern idiom is TaskFlow — plain Python functions decorated into tasks, with return values passed as dependencies via XCom.

Here's a small, idempotent, backfill-safe DAG that waits for a source partition, then loads it — using the partition-overwrite pattern from Chapter 02:

from datetime import datetime
import pendulum
from airflow.decorators import dag, task
from airflow.sensors.external_task import ExternalTaskSensor

@dag(
    schedule="0 2 * * *",                     # daily at 02:00
    start_date=datetime(2026, 1, 1),
    catchup=False,                            # don't auto-run all history on deploy
    max_active_runs=1,                        # serialize runs so partitions don't race
    default_args={"retries": 3, "retry_exponential_backoff": True},
    tags=["orders", "elt"],
)
def orders_pipeline():

    # Deferrable sensor: releases the worker slot while waiting (no worker starvation).
    wait_for_raw = ExternalTaskSensor(
        task_id="wait_for_raw_orders",
        external_dag_id="ingest_raw_orders",
        deferrable=True,                      # async: frees the slot until the trigger fires
        timeout=60 * 60,
        poke_interval=120,
    )

    @task
    def load_partition(data_interval_start=None) -> int:
        # Parameterize on the RUN'S data interval, never on wall-clock "now".
        order_date = data_interval_start.date()
        # Idempotent: this run OWNS order_date's partition and overwrites it.
        return load_orders_partition(order_date)   # delete-then-insert in a txn

    @task
    def run_dbt_marts(rows_loaded: int) -> None:
        run_dbt(select="marts.orders+", vars={"min_rows": rows_loaded})

    wait_for_raw >> run_dbt_marts(load_partition())

orders_pipeline()

The pieces worth being able to explain out loud:

• Data interval, not execution_date. Airflow schedules on interval end: the 02:00 run on the 11th processes the interval that ended then. Binding load_partition to data_interval_start is what makes it a pure function of a date — re-runnable and backfillable.
• Deferrable sensor. A classic sensor occupies a worker slot the entire time it polls; a hundred of them can starve the pool. The deferrable=True / async version hands the wait to a lightweight triggerer and releases the slot — the standard fix for sensor starvation, and a sharp thing to name.
• XCom for control flow, not data. Passing rows_loaded between tasks is fine; XCom has a small size limit (backed by the metadata DB), so never push a dataframe through it — pass a pointer (a path, a partition id) and let the task read the data itself.
• Idempotent tasks + retries. retries=3 with exponential backoff only heals you because load_partition overwrites its partition. Retries on a non-idempotent append would multiply the damage.
• Dynamic task mapping (not shown) lets you fan a task out over a runtime-determined list — e.g. one load task per source region — without hand-writing each one.

Scheduling correctness

Scheduling is where subtle, expensive bugs live, and interviewers probe it precisely because it separates people who've operated pipelines from people who've only written them.

• The interval-end execution model. A daily DAG doesn't run "on" a day — it runs after the day's interval closes, to process the data that accumulated. Confusing the logical date with the wall-clock run time is the root of a whole class of off-by-one partition bugs.
• Timezones & DST. Schedule in an explicit timezone (or UTC) and know what happens across daylight-saving transitions — a naive local schedule can skip or double a run twice a year. Pin it deliberately.
• Late & missed runs. If the scheduler is down, what happens when it recovers? With catchup=True it replays every missed interval; with catchup=False it runs only the latest. Both are correct answers — for different pipelines. Choosing consciously is the signal.

Catchup & backfill

The reason to invest in idempotent, interval-parameterized tasks is that it makes backfill trivial: to rebuild history, you just run the same task across a range of past intervals, each owning and overwriting its partition. No special code path.

• Catchup is automatic history-filling by the scheduler between start_date and now. Leave it off (catchup=False) on deploy unless you actually want the whole backlog to fire — a surprise catchup storm is a common self-inflicted outage.
• Backfill is a deliberate re-run of a date range (fixed a bug, or a source restated history). Because each run overwrites its partition, a backfill converges to the correct state with no double-counting.
• Guardrails. Cap concurrency (max_active_runs, pools) so a 200-day backfill doesn't saturate the warehouse, and be aware that a backfill run and a scheduled run touching the same partition must be serialized. Backfills interact heavily with incremental logic and late data — the full treatment is Chapter 05.

Task-centric vs asset-centric

Chapter 02 introduced the two ways to express a DAG; orchestration is where the choice bites. Task-centric orchestration (classic Airflow) schedules operations — you tell it what to run and wire the edges. Asset-centric orchestration (Dagster's software-defined assets, dbt's model graph) schedules data artifacts — you declare the tables and their dependencies, and the framework derives the graph, tracks staleness, and knows lineage natively.

Airflow has met this world halfway with Datasets (data-aware scheduling): a DAG can be triggered by "dataset X was updated" instead of a clock, so downstream pipelines fire when their inputs are actually ready rather than on a hopeful schedule. That's the task-orchestrator borrowing the best idea from the asset-orchestrators. In an interview, being able to say "I'd move the cross-DAG dependency from a sensor to a dataset trigger" is a concrete maturity signal.

The tool landscape

Be tool-agnostic first, then place the tools. The honest summary: they differ mostly in what they treat as the primitive (task, asset, or durable workflow) and in developer ergonomics — not in whether they can schedule a DAG.

A defensible real-world answer for this company: Airflow (or Dagster) orchestrates ingestion and triggers dbt, dbt owns the transform DAG, and datasets/asset-sensors wire the handoff so marts rebuild when raw actually lands. Naming why you'd combine them beats declaring one tool universally best.

Antipatterns

Leveling ladder

Orchestration maturity is a clean ladder, and it's one interviewers climb quickly once they find your ceiling. Here's how the same "schedule the orders pipeline" answer grows.

Orchestration reliably runs the pipeline — but running it correctly the moment data shows up late is a deeper problem. Next, the first of the three named deep-dive topics: 04 · Late-Arriving Data.

✓ Check yourself

Answer from memory before opening each one.

1. A daily Airflow DAG fires at 02:00 on the 11th. Which day's data should the load task process, and how does the task know?

   Show answer

   The 10th: the run covers the interval that ended at 02:00 on the 11th, and the scheduler hands the task that interval as its data interval. Read the date from the data interval start, never from the wall clock, so a rerun or backfill of any past run builds the right partition. See Airflow, worked.

2. A hundred sensors poll for upstream files and the worker pool is full, so unrelated tasks wait. Why, and what is the fix?

   Show answer

   A classic sensor holds a worker slot for as long as it waits, so many sensors starve the pool. A deferrable sensor hands the wait to the triggerer process and frees the slot until the condition is met. Better still, replace polling with data-aware scheduling where the upstream exposes an event. See Airflow, worked.

3. Marts are scheduled at 03:00 because ingestion "usually finishes by 02:45". One night ingestion takes until 03:20. What does the marts run do, and what do you change?

   Show answer

   It reads a half-built table and publishes wrong numbers, often with every task green. Depend on the data instead: a sensor on the output partition, or a dataset or asset trigger so the marts build when raw actually lands. See scheduling correctness.

4. You deploy a DAG with a start date six months back. With catchup on, what happens, and when would you leave it on anyway?

   Show answer

   The scheduler creates a run for every missed interval at once, which can flood the warehouse and the pool. Leave catchup off by default and backfill deliberately with a concurrency cap. Keep it on for a pipeline where every interval must run in order and the load is cheap, such as a small daily snapshot. See catchup and backfill.

5. Why do idempotent, interval-parameterised tasks make a 90-day backfill an ordinary operation, and what two hazards remain even then?

   Show answer

   Each run owns and overwrites its partition, so running the same task over a past range converges without double counting and needs no special code path. The remaining hazards are load and collision: cap concurrency so the warehouse survives, and serialise a backfill run and a scheduled run that touch the same partition. See catchup and backfill.

6. Your stack is dbt for transforms and ingestion from a dozen APIs. Argue for task-centric or asset-centric orchestration, and say what would change your mind.

   Show answer

   Use both where each fits: asset-aware for the warehouse, because the dbt model graph already is the dependency graph and staleness becomes a first-class question, and task-based for API pulls and training jobs that are not tables. The deciding factors are how much of the work is tables and whether the team needs lineage and staleness from the scheduler. A stack of mostly arbitrary operations tilts task-centric. See task-centric vs asset-centric.

Takeaway

• An orchestrator schedules, resolves dependencies, retries, backfills and surfaces run history; it coordinates work and should not do the compute.
• Hand every task the data interval it owns, so a task is a function of a date and any run can be repeated.
• Retries and backfills are safe only when tasks are idempotent; retrying an append multiplies the damage.
• Airflow runs a daily interval after it closes, so confusing logical date with wall-clock time causes off-by-one partition bugs.
• Depend on data, through sensors, datasets or assets, not on a clock offset that fails silently when upstream runs long.
• Leave catchup off by default, cap concurrency, and treat backfill as a deliberate act.
• Keep logic in dbt or SQL, keep DAG top-level code cheap, and split DAGs by domain.

Orchestration runs the pipeline on time, but data that arrives after its run is a separate problem with its own mechanisms. → Late-Arriving Data

Further reading

• Airflow: DAG Runs — Apache Airflow docs. Data intervals, logical date, catchup and backfill semantics in the vendor's words.
• Airflow: Best Practices — Apache Airflow docs. Top-level code cost, idempotent tasks, XCom limits and the other traps listed in this chapter.
• Airflow: Asset-Aware Scheduling — Apache Airflow docs. Data-aware triggers, which earlier 2.x releases call datasets.
• Dagster: Assets — Dagster docs. The asset-centric model, with the graph derived from declared data dependencies.
• Prefect: Flows — Prefect docs. The dynamic, Python-first flow model, for comparison with DAG declaration.