Section B · DE using neoclouds

Orchestration Across Providers

How DE pipelines coordinate data work and GPU jobs across multiple providers. Orchestrator choice, multi-provider patterns, retry semantics on spot, and the operational discipline that keeps everything coherent.

Orchestrator landscape

The orchestrators most commonly used in AI / GPU-cloud DE work:

Airflow

The veteran. Mature, broadly known, extensive operator ecosystem. Limitations: weaker data-asset abstractions; mixed support for streaming and ML-specific patterns. Strong fit for traditional ETL with periodic ML training jobs as one node.

Dagster

Asset-oriented. Treats datasets, models, and downstream artifacts as first-class assets. Strong fit for ML / DE workflows where lineage matters. Growing rapidly through 2024-2026.

Prefect

Pythonic, dynamic workflow definition. Good developer experience. Adoption strongest in mid-market.

Argo Workflows

Kubernetes-native. Often paired with Kubeflow for ML pipelines. Strong fit when your stack is already K8s-heavy.

Flyte

Kubernetes-native and ML-pipeline-oriented. Strong type system; reproducibility features; growing adoption.

Custom job submission

Many AI organizations build a thin job-submission layer over their primary orchestrator. The orchestrator schedules and tracks; the custom layer translates a job request into provider-specific API calls.

Multi-provider patterns

Common patterns for handling multiple GPU providers from one orchestrator:

Provider as parameter

The DAG / workflow takes a "provider" parameter. Branching logic submits the actual GPU job to the chosen provider. Simple; works for small numbers of providers; brittle as providers' surfaces diverge.

Provider abstraction layer

A thin internal library that abstracts "submit GPU job" across providers. The orchestrator only calls the abstraction; the library handles provider-specific API translation. More maintenance but cleaner.

Kubernetes federation

Run K8s clusters at each provider; treat them as labeled targets. Orchestrator submits to a federated control plane that routes to clusters. Adds operational complexity; powerful for heavy users.

External scheduling service

A service that takes job specs, looks at current provider availability and pricing, and chooses where to submit. The orchestrator submits to the service; the service handles the routing decision.

Most shops start with parameter-based or abstraction-layer and graduate to more sophisticated patterns as they grow.

Provisioning ephemeral GPU clusters

For training jobs, you often need to spin up a multi-node GPU cluster, run the job, tear it down. The DE-relevant aspects:

Cluster startup time. 5-30 minutes from request to ready, depending on provider and configuration. Plan for it.
Image / environment management. Container image with CUDA, drivers, frameworks pre-baked saves boot time.
Storage mounts. Get the storage attached at boot; don't fall back to copying data after.
Network setup. InfiniBand or other fast interconnect configured.
Job submission inside cluster. Slurm, Ray, or custom; some providers manage this.
Teardown discipline. Tear down promptly when done; idle GPU clusters burn money fast.

Idempotency & retries

Failure handling is a first-class concern. Retries that aren't idempotent cause duplicate billing, duplicate training, corrupted outputs.

Job-level retries. The orchestrator retries the job. Job submission should be idempotent — same inputs → same output, no side effects from duplicate runs.
Step-level retries. Within a job, individual data loading or training steps may retry. Checkpointing is essential.
Run identifiers. Every job gets a deterministic ID derived from inputs. Duplicate submissions de-dupe to the same logical run.
Output staging. Write to staging locations, then atomic rename / publish. Avoids partial outputs being mistaken for complete.

The classic DE idempotency mantra applies, just at higher stakes because each retry costs real GPU money.

Spot & preemption

Spot / interruptible / preemptible GPU instances offer significant cost savings (often 50% off on-demand) at the cost of being killable.

To use them well:

Frequent checkpointing — at least every N minutes, where N is acceptable rework on preemption.
Quick restart — the orchestrator should resubmit on a different instance promptly.
Preemption-aware code — training code that handles SIGTERM gracefully.
Budget for some wasted compute — preemption mid-step means losing the partial step.
Don't use spot for time-critical jobs — like a job blocking a model release.

Observability

DE pipelines on neoclouds need observability across:

Orchestrator runs (success/fail, durations).
GPU job-level metrics (utilization, memory, throughput).
Storage I/O.
Network bandwidth.
Cost per run.
Output quality (does the resulting model / dataset pass eval).

This data ends up in the warehouse via the orchestrator's tracking tables, GPU metrics agents, and the provider's billing exports. Stitching it together for cost attribution (see next chapter) is itself a DE pipeline.

✓ Check yourself

Answer from memory before opening each one.

The orchestrator times out waiting on a GPU job submission and retries, but the first submission had actually started. What goes wrong, and how do you prevent it?

Show answer
You run two clusters, pay twice, and both may write to the same output. Derive a deterministic run ID from the inputs so a duplicate submission resolves to the same logical run, and write outputs to staging with an atomic publish. See idempotency and retries.
A job checkpoints every 30 minutes and its spot instance is preempted about every 4 hours. Assume lost work averages half the interval and a restart takes 10 minutes. Roughly what does the discount cost you?

Show answer
About 25 minutes lost per 4 hours, near 10 percent more compute. At the 50 percent discount the chapter cites, effective cost is around 55 percent of on-demand, so spot still wins. It stops winning if the job blocks a release. See spot and preemption.
You orchestrate two providers with a provider parameter in the DAG. When does that design start to hurt, and what replaces it?

Show answer
When provider surfaces diverge, so branching logic spreads through the DAGs and each new feature needs another branch. Move the provider-specific calls behind an internal abstraction library, and later perhaps a scheduling service. See multi-provider patterns.
A multi-node training job needs 8 nodes for 2 hours, and cluster startup takes 5 to 30 minutes. What do you do in the pipeline to avoid paying for waiting?

Show answer
Bake CUDA, drivers and frameworks into the image, attach storage at boot instead of copying data afterward, and tear down promptly when the job ends. Plan the startup window in the schedule and in the cost estimate. See provisioning ephemeral clusters.
A team cares most about dataset and model lineage and is choosing between Airflow and Dagster. How do you decide?

Show answer
Dagster treats datasets and models as first-class assets, which fits lineage-heavy work. Airflow wins when you already run a large estate with its operator ecosystem and lineage is secondary. Weigh the asset model against the cost of migration. See the orchestrator landscape.

Takeaway

The orchestrator schedules and tracks, and a thin job-submission layer translates a request into provider-specific calls.
Start with provider-as-parameter or an abstraction library, and move to federation or a scheduling service only when scale demands it.
Retries cost GPU money, so submissions use deterministic run IDs and outputs are staged and published atomically.
Spot is worth it when checkpoints are frequent and restarts are quick, and it is wrong for jobs that block a release.
Cluster startup of 5 to 30 minutes and idle teardown belong in the cost and schedule model.
Orchestrator tables, GPU metrics and billing exports must join so each run has a cost.

Orchestration produces the runs, and cost engineering makes their price visible. → Cost Engineering