Section B · DS using neoclouds

The DS Workflow on GPU Clouds

How a DS / ML workflow shifts when the compute runs on neocloud infrastructure rather than hyperscaler default. The shape of the work, the cycle from experiment to production, and the operational realities.

The shape of DS work

In a 2026 AI / ML organization running on neoclouds, DS / ML work has these characteristic activities:

Research and experimentation. Trying ideas — new architectures, data mixes, training regimes — usually at small scale.
Fine-tuning at scale. Customizing open-source or in-house models on customer-specific data.
Large-scale training. The big training runs (full pre-train or significant fine-tunes).
Evaluation. Building eval sets, running benchmarks, tracking metric movement.
Production deployment. Serving models to customers via APIs.
Monitoring. Watching production model behavior and feeding back insights.

Each activity has different infrastructure needs. Research uses cheap interruptible GPUs; large-scale training uses dedicated clusters; production inference uses either dedicated or per-token managed services.

The experiment-to-production cycle

The standard cycle:

Hypothesis: a problem worth solving or an idea worth testing.
Quick experimentation at small scale (single GPU or small cluster). Often on marketplaces (Vast, RunPod Community) for cost.
Iteration on the most promising approaches.
Scale up to a full training or fine-tune. Often on dedicated providers (CoreWeave, Crusoe, Lambda, Together training).
Evaluation against the eval set; iteration if needed.
Deployment to inference infrastructure (Together API, Hyperbolic inference, RunPod Serverless, or self-hosted on dedicated).
Production monitoring and feedback into the next cycle.

The cycle's velocity is shaped by how friction-free each step is. Friction at the small-experiment step (waiting for a GPU; provider issues) slows iteration. Friction at the production step (deployment complexity; cost) slows delivery.

How neoclouds change the workflow

Compared to a hyperscaler-default workflow:

Cheaper experimentation. Marketplaces dramatically lower the cost of trying things. More ideas tested.
More variability. Marketplace supply quality varies; planners have to allow for it.
Better fabric available. Dedicated neoclouds have InfiniBand cluster fabric that's cheaper than hyperscaler equivalents.
Different commercial structure. Reserved-capacity contracts for large training; per-token APIs for inference.
Multi-provider posture. Most teams don't lock to one provider.
Sharper cost focus. Compute is a bigger fraction of total spend than at most hyperscaler customers.

Roles and handoffs

In a mature org, roles partition like this:

Research / applied DS: Hypotheses, experiments, evaluation, model improvement.
ML engineers / infrastructure: Pipelines, training infrastructure, serving infrastructure, monitoring.
Data engineering: Upstream and downstream data pipelines.
Platform / FinOps: Cost reporting, provider relationships, reservation negotiation.

Smaller teams blur these. The DS who knows enough infrastructure to spin up a cluster and run a fine-tune unassisted is valuable; the DS who can't speak to provider economics is increasingly limited.

✓ Check yourself

Answer from memory before opening each one.

Your team's experiments feel slow even though GPUs are cheap and plentiful. Where in the cycle do you look first, and what do you expect to find at each end?

Show answer
Look at the friction in each step, not the price. At the small-experiment end the cost is waiting for a GPU or fighting a provider issue, which slows iteration. At the production end it is deployment complexity and cost, which slows delivery. Cheap compute does not help if the loop around it is slow. See the cycle.
A single org runs a marketplace GPU for ideas, a reserved cluster for a big fine-tune and a per-token API for serving. Why is that not inconsistency?

Show answer
Each activity has a different need. Research wants cheap, interruptible GPUs, large training wants dedicated clusters with fast fabric, and inference wants per-token or dedicated serving. One provider is rarely best at all three, so most teams stay multi-provider. See the shape of DS work and how neoclouds change the workflow.
Marketplaces cut the cost of trying an idea, but the chapter warns that supply quality varies. How does that change the way you plan a project that depends on marketplace compute?

Show answer
Treat the variability as a planning input. Budget for restarts, keep runs small enough that a lost node costs little, and keep work with a firm delivery date off the cheapest tier. Cheaper experiments buy you more ideas tested, not a guaranteed schedule. See how neoclouds change the workflow.
A two-person team asks whether the DS should learn to launch and tear down clusters, or leave that to an ML engineer they do not have. What do you advise?

Show answer
Learn it. In a mature org the roles partition into research, ML engineering, data engineering and FinOps, but a small team blurs them. A DS who can run a fine-tune unassisted and can discuss provider economics is more useful than one who cannot. At larger scale, hand the platform work to ML engineers and keep the DS focus on hypotheses and evaluation. See roles and handoffs.
Compute is a larger share of spend at a neocloud customer than at a typical hyperscaler customer. Name two habits in a DS workflow that follow from that.

Show answer
Cost becomes a design input: you compare providers per activity, and you size experiments before launching them. Both follow from compute being a large line in the budget, so each wasted run is visible. See how neoclouds change the workflow.

Takeaway

DS work on a neocloud splits into research, fine-tuning, large training, evaluation, deployment and monitoring, and each needs different infrastructure.
The cycle from hypothesis to production is only as fast as its most frictionful step, whether that is waiting for a GPU or deploying a model.
Marketplaces make experiments cheap but variable; dedicated clusters give fast fabric at contract terms; per-token APIs suit serving.
Most teams use several providers, because no single provider is best at every activity.
Compute is a large share of spend, so cost awareness is part of the DS job, not a finance concern.
In small teams the DS also needs enough infrastructure skill to run a fine-tune unassisted.

Provider choice from the DS side is the first decision in that cycle. → Picking the Right Provider for ML Work