Section B · DE using neoclouds

The DE Landscape for AI Workloads

Where data engineering sits in an AI-first organization that runs on neocloud infrastructure. The work that has to be done before ML happens, alongside ML, and after ML deploys.

Where DE fits

In a 2026 AI company running serious training and inference, the data-engineering function sits at several intersections:

Upstream of training. Pulling raw data from sources (web crawls, customer data, partner feeds, internal logs), cleaning, deduplicating, format-converting, sharding for parallel reading by training jobs.
Alongside fine-tuning and evaluation. Building eval datasets, tracking provenance, version-controlling data that gets fed to fine-tunes.
Downstream of inference. Capturing prompts and responses for analytics, building feedback loops, persisting outputs into product systems.
Around cost and observability. Pipelines that pull GPU billing data, compute-cost-per-feature, attribute spend to teams and experiments.

At smaller AI startups, one person may cover all four. At larger AI companies, these are distinct teams: data infrastructure, ML data ops, ML evaluation, and FinOps / platform analytics.

Shape of the work

The shape of DE work changes when GPU clouds are in the picture:

1. Data movement matters more

In traditional data warehouse work, you move data between warehouses. In AI / GPU-cloud work, you move data into GPU compute — often across providers, often at TB-to-PB scale, often against tight time windows when a GPU cluster is booked.

"Get the dataset to the training cluster before the cluster boots" is a real DE problem. So is "do it without paying for 10 idle GPU-hours waiting for the data."

2. Compute is the dominant cost

Storage and warehouse compute matter; GPU compute matters more. A single training run might consume more cloud spend in 48 hours than a quarter of warehouse queries. DE work has to be tuned with that cost ratio in mind.

3. Failures are routine

Spot preemption, node failures, network partitions, OOMs in GPU memory — these happen routinely. Pipelines have to assume restart-from-checkpoint as the default, not the exception.

4. Provenance is foundational

Training data provenance — what raw data, what cleaning, what version — is increasingly tied to compliance, licensing, and reproducibility. DE owns the chain.

5. Eval and feedback loops are first-class

The DE work that supports evaluation (golden sets, drift detection, output capture, labeling pipelines) is on the same priority tier as training-data pipelines.

The handoff to ML

The classic DE/ML handoff in an AI / GPU-cloud organization:

DE produces a versioned, partitioned, sharded dataset on shared storage.
ML team picks the dataset version for a training run.
Training job is submitted against a GPU cluster (provisioned via neocloud).
Job reads dataset from shared storage; writes checkpoints back; emits metrics.
DE captures the run metadata (dataset version, hyperparameters, cluster, cost).
Model artifact gets versioned; DE pipelines support downstream serving.

The handoff is tighter than in classic analytics work. DE can't ship a dataset and walk away — the training job's success depends on details (format, partitioning, throughput) that the DE controlled. DE and ML teams often share oncall coverage.

Cross-cutting concerns

Several concerns thread through all DE work on neoclouds:

Multi-provider posture. Most serious AI companies use more than one neocloud. DE has to design for it.
Cost accountability. Compute spend has to be attributable to experiments, teams, customers. The pipelines that do this are DE work.
Data licensing and provenance. Regulatory and contractual constraints on what data can train what models.
Security at the boundary. GPU instances are sometimes semi-trusted (especially on marketplace neoclouds). Secrets management, encryption, and data residency need to be thought through.
Reproducibility. Models that can't be reproduced (because the underlying data was lost or mutated) are a liability.

✓ Check yourself

Answer from memory before opening each one.

A training job on a booked 64-GPU cluster sits idle for its first 40 minutes. The data team says the dataset was delivered the day before. Why is this still a data engineering problem?

Show answer
Because format, partitioning and read throughput decide whether the job can start fast, and data engineering chose all three. The job's success depends on those details, so the handoff is tighter than in analytics and the two teams often share on-call. The cost is 64 GPUs times 40 minutes of nothing. See the handoff to ML.
A warehouse batch job that fails can alert and rerun in the morning. Why do pipelines around GPU work assume restart from a checkpoint as the default?

Show answer
Spot preemption, node failures, network partitions and GPU out-of-memory errors are routine, and each restart from zero throws away paid GPU hours. The unit of loss is a training run, not a query. Design for resume first and treat a clean run as the lucky case. See shape of the work.
A colleague calls training-data lineage bookkeeping that can wait until after launch. What three pressures make that wrong?

Show answer
Compliance, licensing and reproducibility. Contracts and regulation limit which data may train which model, and a model whose data was lost or changed cannot be reproduced, which makes it a liability. Lineage is cheap to record at build time and often impossible to rebuild later. See provenance and cross-cutting concerns.
Your company plans an inference pilot on a marketplace provider instead of a dedicated one. Which concern changes first, and what do you do about it?

Show answer
Security at the boundary. Marketplace hosts are semi-trusted, so keep secrets out of images, encrypt data at rest, mount only the data the job needs and keep regulated data off them. The pipeline design is the same; the trust assumption is not. See cross-cutting concerns.
A 15-person AI startup hires its second data engineer. One person has been covering upstream data, evaluation data, inference capture and cost pipelines. Which area do you hand over first?

Show answer
There is no single answer. Cost and observability usually pay back soonest because compute dominates spend and untagged jobs cannot be fixed later. Hand over upstream training data first if GPUs sit idle waiting for it. Pick the area with the largest measured waste. See where DE fits.

Takeaway

Data engineering for AI workloads is ordinary data engineering tilted by four forces: compute-dominant cost, routine failure, tight ML handoff and mandatory provenance.
Data has to move into GPU compute, often across providers and at terabyte scale, so idle GPU hours waiting for data are a data engineering defect.
The handoff to ML is tighter than in analytics because format, partitioning and throughput decide whether training succeeds, so the teams share on-call.
Pipelines assume restart from checkpoint, because preemption and node failure are normal.
Provenance, cost attribution and reproducibility are owned by data engineering and thread through every other topic in this guide.

The first decision that shapes everything else is where the work runs. → Choosing a Provider for Data Pipelines