Orchestration Across Providers

Orchestrator landscape

The orchestrators most commonly used in AI / GPU-cloud DE work:

Airflow

The veteran. Mature, broadly known, extensive operator ecosystem. Limitations: weaker data-asset abstractions; mixed support for streaming and ML-specific patterns. Strong fit for traditional ETL with periodic ML training jobs as one node.

Dagster

Asset-oriented. Treats datasets, models, and downstream artifacts as first-class assets. Strong fit for ML / DE workflows where lineage matters. Growing rapidly through 2024-2026.

Prefect

Pythonic, dynamic workflow definition. Good developer experience. Adoption strongest in mid-market.

Argo Workflows

Kubernetes-native. Often paired with Kubeflow for ML pipelines. Strong fit when your stack is already K8s-heavy.

Flyte

Kubernetes-native and ML-pipeline-oriented. Strong type system; reproducibility features; growing adoption.

Custom job submission

Many AI organizations build a thin job-submission layer over their primary orchestrator. The orchestrator schedules and tracks; the custom layer translates a job request into provider-specific API calls.

Multi-provider patterns

Common patterns for handling multiple GPU providers from one orchestrator:

Provider as parameter

The DAG / workflow takes a "provider" parameter. Branching logic submits the actual GPU job to the chosen provider. Simple; works for small numbers of providers; brittle as providers' surfaces diverge.

Provider abstraction layer

A thin internal library that abstracts "submit GPU job" across providers. The orchestrator only calls the abstraction; the library handles provider-specific API translation. More maintenance but cleaner.

Kubernetes federation

Run K8s clusters at each provider; treat them as labeled targets. Orchestrator submits to a federated control plane that routes to clusters. Adds operational complexity; powerful for heavy users.

External scheduling service

A service that takes job specs, looks at current provider availability and pricing, and chooses where to submit. The orchestrator submits to the service; the service handles the routing decision.

Most shops start with parameter-based or abstraction-layer and graduate to more sophisticated patterns as they grow.

Provisioning ephemeral GPU clusters

For training jobs, you often need to spin up a multi-node GPU cluster, run the job, tear it down. The DE-relevant aspects:

• Cluster startup time. 5-30 minutes from request to ready, depending on provider and configuration. Plan for it.
• Image / environment management. Container image with CUDA, drivers, frameworks pre-baked saves boot time.
• Storage mounts. Get the storage attached at boot; don't fall back to copying data after.
• Network setup. InfiniBand or other fast interconnect configured.
• Job submission inside cluster. Slurm, Ray, or custom; some providers manage this.
• Teardown discipline. Tear down promptly when done; idle GPU clusters burn money fast.

Idempotency & retries

Failure handling is a first-class concern. Retries that aren't idempotent cause duplicate billing, duplicate training, corrupted outputs.

• Job-level retries. The orchestrator retries the job. Job submission should be idempotent — same inputs → same output, no side effects from duplicate runs.
• Step-level retries. Within a job, individual data loading or training steps may retry. Checkpointing is essential.
• Run identifiers. Every job gets a deterministic ID derived from inputs. Duplicate submissions de-dupe to the same logical run.
• Output staging. Write to staging locations, then atomic rename / publish. Avoids partial outputs being mistaken for complete.

The classic DE idempotency mantra applies, just at higher stakes because each retry costs real GPU money.

Spot & preemption

Spot / interruptible / preemptible GPU instances offer significant cost savings (often 50% off on-demand) at the cost of being killable.

To use them well:

• Frequent checkpointing — at least every N minutes, where N is acceptable rework on preemption.
• Quick restart — the orchestrator should resubmit on a different instance promptly.
• Preemption-aware code — training code that handles SIGTERM gracefully.
• Budget for some wasted compute — preemption mid-step means losing the partial step.
• Don't use spot for time-critical jobs — like a job blocking a model release.

Observability

DE pipelines on neoclouds need observability across:

• Orchestrator runs (success/fail, durations).
• GPU job-level metrics (utilization, memory, throughput).
• Storage I/O.
• Network bandwidth.
• Cost per run.
• Output quality (does the resulting model / dataset pass eval).

This data ends up in the warehouse via the orchestrator's tracking tables, GPU metrics agents, and the provider's billing exports. Stitching it together for cost attribution (see next chapter) is itself a DE pipeline.

Takeaway

Orchestration is where the DE work crosses the boundary into operations. Multi-provider abstraction, idempotency discipline, and graceful preemption handling separate teams that operate efficiently from teams that bleed GPU money. The next chapter goes deep on cost engineering — the discipline that makes the difference visible.