Section B · DS using neoclouds · Critical chapter

Picking the Right Provider for ML Work

Provider choice from the ML / DS perspective. Workload-by-workload guidance for matching the right neocloud to the right ML activity.

DS-side decision differs from DE-side

The DE guide's provider chapter framed the decision around data movement, orchestration, and cost engineering. The DS perspective adds:

Time-to-result matters as much as cost (iteration velocity).
Hardware-specific performance matters (some workloads need specific SKUs).
Tooling fit matters (does the provider's environment support your framework cleanly).
Quality vs cost tradeoff differs by ML activity (research tolerates flakiness; production doesn't).

Research / experimentation

For exploratory ML work — quick training runs to test ideas, evaluation on a candidate model, debugging — the priority is cheap GPUs available immediately.

First choice: Vast.AI for the lowest prices and broadest hardware selection.
Second choice: RunPod Community Cloud for slightly more polish at modest premium.
Third choice: Hyperbolic marketplace or TensorDock for alternative supply.
Avoid: Reserved-capacity providers — overkill and slow procurement.

Interruptible / spot instances are ideal here. Lost work is small; cost savings are large.

Fine-tuning

For fine-tuning small-to-mid models (sub-70B, LoRA / QLoRA / full):

Single-GPU or small multi-GPU: Marketplaces or RunPod Secure / Lambda.
Larger fine-tunes (8-32 GPUs): Lambda, RunPod Secure, Together training clusters.
Managed fine-tuning: Together.AI's fine-tuning service for if you want zero infrastructure work.

The choice depends on whether you want to manage the infrastructure (cheaper, more control) or pay a premium for managed (faster delivery).

Large-scale training

For training larger models from scratch or significant pre-train extensions (100+ GPUs in cluster):

Top tier: CoreWeave, Crusoe, Nebius — multi-thousand-GPU cluster fabric, NVIDIA reference designs.
Mid tier: Lambda's 1-Click Cluster for hundreds-of-GPU scale without multi-year contracts.
Avoid: Marketplaces (no fabric for distributed training).

For multi-thousand-GPU training, reservations months in advance are typically required. This is procurement work as much as DS work.

Inference deployment

Production inference options:

Managed per-token API: Together.AI, Hyperbolic inference, Fireworks. Best for spiky traffic and lower volumes.
Dedicated endpoints: Together's dedicated, RunPod Serverless dedicated. For predictable medium-volume.
Self-hosted on dedicated GPUs: Lambda, RunPod Secure, CoreWeave for high-volume / custom models / control needs.
Spot / marketplace inference: For non-production or fault-tolerant batch inference workloads.

The break-even between managed and self-hosted depends on utilization. Below ~50% utilization, managed wins. Above ~70%, self-hosted wins. The middle is a judgment call.

Evaluation infrastructure

Running eval suites at scale (especially for LLMs with many benchmarks) is its own workload class:

Often parallelizable across many smaller GPUs.
Latency-tolerant (run overnight; check results in morning).
Spot-friendly.

Marketplaces are well-suited. Some teams build dedicated eval clusters on Lambda or RunPod Secure for repeatability.

✓ Check yourself

Answer from memory before opening each one.

A colleague wants to run a 256-GPU pre-train extension on the cheapest marketplace GPUs and says the hourly rate is far lower. What do you tell them?

Show answer
Marketplaces have no fabric for distributed training. Gradient exchange across nodes is limited by the interconnect, so cheap GPUs sit idle waiting on the network and the real cost per step is higher. Use a dedicated provider with a cluster fabric. See large-scale training.
A model serves about 5 requests a second on average with a 10-minute burst to 40 each afternoon. Do you choose a managed per-token API or self-hosted GPUs?

Show answer
Choose the managed API. Self-hosted GPUs sized for the burst sit mostly idle, so utilization is well below the roughly 50% break-even and you pay for idle capacity. Revisit when average load rises. See inference deployment.
The same workload runs at about 60% utilization. Managed or self-hosted?

Show answer
It is a judgment call, as the break-even band is roughly 50% to 70%. Lean to self-hosted if traffic is growing, the model is custom or latency control matters and you have people to run it. Lean to managed if traffic is flat, you are short of operations staff or you want zero infrastructure work. See inference deployment.
Name two criteria the DS side adds to a provider decision that the data-engineering side does not weigh as heavily.

Show answer
Time-to-result, because iteration velocity matters as much as cost, and hardware fit, because some workloads need a specific SKU. Tooling fit and how much flakiness the activity tolerates also count. See how the DS decision differs.
Evaluation runs are parallel, latency-tolerant and spot-friendly, so why do some teams still build a dedicated eval cluster?

Show answer
Repeatability. When you compare model versions you want the same hardware and software for each run, and marketplace supply varies from run to run. A dedicated cluster removes that source of noise from the comparison. See evaluation infrastructure.
A fine-tune of a sub-70B model will use 16 GPUs. What decides whether you run it yourself on Lambda or RunPod Secure or use a managed fine-tuning service?

Show answer
Whether you want control and a lower price or faster delivery with no infrastructure work. Self-managed is cheaper and flexible, and managed costs a premium. Staff time and deadline usually decide it. See fine-tuning.

Takeaway

There is no single right provider; the skill is matching each ML activity to a provider class.
Research and evaluation want cheap, interruptible GPUs on marketplaces, where lost work is small and savings are large.
Distributed training needs a dedicated provider with cluster fabric; marketplaces cannot supply it.
Fine-tuning splits on one choice: manage the infrastructure for lower cost, or pay for a managed service for speed.
For inference, managed APIs win below roughly 50% utilization and self-hosting wins above roughly 70%.
Multi-thousand-GPU training needs reservations months ahead, which makes it procurement work as much as DS work.

Once the provider class is chosen, the work becomes running training and fine-tuning on it. → Training & Fine-Tuning Workflows