Section B · DE using neoclouds

The DE Landscape for AI Workloads

Where data engineering sits in an AI-first organization that runs on neocloud infrastructure. The work that has to be done before ML happens, alongside ML, and after ML deploys.

Where DE fits

In a 2026 AI company running serious training and inference, the data-engineering function sits at several intersections:

  • Upstream of training. Pulling raw data from sources (web crawls, customer data, partner feeds, internal logs), cleaning, deduplicating, format-converting, sharding for parallel reading by training jobs.
  • Alongside fine-tuning and evaluation. Building eval datasets, tracking provenance, version-controlling data that gets fed to fine-tunes.
  • Downstream of inference. Capturing prompts and responses for analytics, building feedback loops, persisting outputs into product systems.
  • Around cost and observability. Pipelines that pull GPU billing data, compute-cost-per-feature, attribute spend to teams and experiments.

At smaller AI startups, one person may cover all four. At larger AI companies, these are distinct teams: data infrastructure, ML data ops, ML evaluation, and FinOps / platform analytics.

Shape of the work

The shape of DE work changes when GPU clouds are in the picture:

1. Data movement matters more

In traditional data warehouse work, you move data between warehouses. In AI / GPU-cloud work, you move data into GPU compute — often across providers, often at TB-to-PB scale, often against tight time windows when a GPU cluster is booked.

"Get the dataset to the training cluster before the cluster boots" is a real DE problem. So is "do it without paying for 10 idle GPU-hours waiting for the data."

2. Compute is the dominant cost

Storage and warehouse compute matter; GPU compute matters more. A single training run might consume more cloud spend in 48 hours than a quarter of warehouse queries. DE work has to be tuned with that cost ratio in mind.

3. Failures are routine

Spot preemption, node failures, network partitions, OOMs in GPU memory — these happen routinely. Pipelines have to assume restart-from-checkpoint as the default, not the exception.

4. Provenance is foundational

Training data provenance — what raw data, what cleaning, what version — is increasingly tied to compliance, licensing, and reproducibility. DE owns the chain.

5. Eval and feedback loops are first-class

The DE work that supports evaluation (golden sets, drift detection, output capture, labeling pipelines) is on the same priority tier as training-data pipelines.

The handoff to ML

The classic DE/ML handoff in an AI / GPU-cloud organization:

  1. DE produces a versioned, partitioned, sharded dataset on shared storage.
  2. ML team picks the dataset version for a training run.
  3. Training job is submitted against a GPU cluster (provisioned via neocloud).
  4. Job reads dataset from shared storage; writes checkpoints back; emits metrics.
  5. DE captures the run metadata (dataset version, hyperparameters, cluster, cost).
  6. Model artifact gets versioned; DE pipelines support downstream serving.

The handoff is tighter than in classic analytics work. DE can't ship a dataset and walk away — the training job's success depends on details (format, partitioning, throughput) that the DE controlled. DE and ML teams often share oncall coverage.

Cross-cutting concerns

Several concerns thread through all DE work on neoclouds:

  • Multi-provider posture. Most serious AI companies use more than one neocloud. DE has to design for it.
  • Cost accountability. Compute spend has to be attributable to experiments, teams, customers. The pipelines that do this are DE work.
  • Data licensing and provenance. Regulatory and contractual constraints on what data can train what models.
  • Security at the boundary. GPU instances are sometimes semi-trusted (especially on marketplace neoclouds). Secrets management, encryption, and data residency need to be thought through.
  • Reproducibility. Models that can't be reproduced (because the underlying data was lost or mutated) are a liability.

Takeaway

Data engineering for AI workloads on neoclouds is recognizably DE work, but tilted: compute-dominant economics, multi-provider posture, tight ML handoff, and routine failure handling. The next chapter starts the practitioner playbook with provider selection.