Choosing a Provider for Data Pipelines

Decision axes

Seven dimensions matter most for a DE evaluating providers:

1. Reliability and SLA. Whether jobs run end-to-end without intervention. Critical for production; tolerable for research.
2. Network / cluster fabric. Whether you can run multi-node training. Required for serious training; irrelevant for single-node fine-tuning.
3. Storage architecture. Whether the provider has fast, scalable storage adjacent to the GPUs. Matters enormously for training-data throughput.
4. Pricing model. Per-second on-demand vs reserved vs spot. Each shapes how you design pipelines.
5. Geography and data residency. Where the GPUs live relative to your data sources. Latency and compliance.
6. Compliance posture. SOC 2, HIPAA, FedRAMP. Required for some regulated workloads.
7. Integration surface. APIs, CLI, orchestrator support. How well it fits your stack.

Workload-to-provider map

Mapping workload types to provider classes (covered in the Neocloud topic guides):

Data-specific considerations

From the DE's perspective specifically — beyond the general workload fit — these matter:

Storage adjacency

How fast can you get TBs of training data to the GPU? Providers vary:

• CoreWeave, Crusoe, Nebius offer fast parallel storage in-cluster.
• Lambda has object storage and persistent volumes with reasonable performance.
• Marketplaces typically only have local NVMe; bring-your-own-storage at the network layer.

If your training reads 50 TB from S3 every job, you're either paying for slow loading or paying for fast loading. Pick the provider accordingly.

Egress and cross-cloud

Moving training data into the GPU cluster from external storage (your S3, your customer's warehouse) incurs egress. Where the data starts and where the GPU lives shape this cost.

Per-second vs per-hour billing

Per-second billing (RunPod, modern neoclouds) means pipeline startup overhead is cheap; per-hour billing (some older providers) means you should batch work to amortize startup.

Spot / interruptible handling

If you use spot or interruptible instances, your pipeline has to handle preemption gracefully. Some providers give better preemption notice than others.

Multi-tenant isolation

Marketplace providers have weaker isolation than dedicated. If your data is sensitive, encrypt at rest and don't use marketplaces for the sensitive workloads.

Multi-provider as default

Most serious AI organizations operate across multiple providers. Reasons:

• Capacity constraints — no single provider has all the H200s you might need at the moment you need them.
• Negotiating leverage — single-provider lock-in gives the provider pricing power.
• Geographic distribution — different providers cover different regions.
• Cost optimization — different providers are cheapest for different SKUs at different times.
• Resilience — outages happen.

DE has to design for this. Abstracting over providers — through orchestrator-level adapters, through Kubernetes operators, through custom job submission services — is foundational work.

Contracting and commitments

Reserved-capacity contracts are usually negotiated by procurement / leadership, but DE input matters:

• What hardware mix you actually need over the term.
• What flexibility (region, scaling, GPU type changes) is required.
• What technical integration commitments the vendor is making (storage, networking, orchestration support).
• Exit and migration provisions if the relationship doesn't work.

The DE who builds the multi-provider abstraction can also help leadership understand exactly what they're committing to and what's reversible.

A working decision template

A reusable checklist when evaluating a new provider:

1. What's the primary workload class this provider will serve?
2. What's the cost gap to alternatives (on-demand list, reserved, after negotiation)?
3. What's the storage architecture and how does it match our data layout?
4. What's the network fabric and does it match our cluster sizes?
5. What's the orchestrator integration story?
6. What's the compliance posture and does it cover our workloads?
7. What's the support model (response times, escalation paths)?
8. What's the migration cost in if it doesn't work?
9. How does it fit into the multi-provider posture we already have?
10. What does the renewal / pricing-change story look like?

✓ Check yourself

Answer from memory before opening each one.

1. Your training job reads 50 TB from object storage on every run, and the candidate provider offers only local NVMe. What are your two ways to pay, and which provider traits change the answer?

   Show answer

   You pay in idle GPU time while data loads slowly, or in staging time and egress to load it fast. Providers with fast parallel storage inside the cluster remove most of that choice, while NVMe-only hosts leave it with you. Price the loading path along with the GPU hour. See storage adjacency.

2. Why does per-second versus per-hour billing change how you design pipeline startup?

   Show answer

   With per-second billing, startup overhead costs only the seconds it takes, so many short jobs are fine. With per-hour billing, every partial hour is paid in full, so you batch work to fill the hours. The billing grain sets the right job size. See per-second vs per-hour billing.

3. A team wants to fine-tune on customer health records using the cheapest marketplace listing. What do you say, and what do you ask first?

   Show answer

   Say no for that data. Marketplaces have weaker isolation and generally sit in the avoid column for regulated workloads. Ask which compliance regime applies, such as HIPAA, and whether a hyperscaler or a contracted dedicated provider covers it. Encrypt at rest, and keep marketplaces for non-sensitive jobs. See the workload map.

4. Why is multi-provider the default for serious AI organizations, and what does that demand from data engineering?

   Show answer

   Capacity, negotiating leverage, geography, cost differences by SKU and resilience each argue against a single vendor. It demands an abstraction over providers, such as orchestrator adapters, Kubernetes operators or a submission service, so jobs do not hard-code one provider's API. See multi-provider as default.

5. Procurement offers a 30 percent discount for a three-year reservation with one provider. What does data engineering add to that decision?

   Show answer

   Facts about the term: the hardware mix you will need, how much room you need to change region, scale or GPU type, what integration the vendor commits to, and what exit costs. The discount wins only if your utilization stays above the commitment. Show which parts are reversible. See contracting and commitments.

Takeaway

• Provider choice is made per workload class, not once: large training, fine-tuning, production inference and research each have a different best fit.
• Seven axes decide it: reliability, fabric, storage, pricing model, geography, compliance and integration surface.
• Storage adjacency and egress often cost more than the GPU-hour price difference, so price the data path with the compute.
• Marketplace hosts suit research and cost-sensitive work, not regulated or sensitive data.
• Multi-provider is the default, which makes a provider abstraction foundational work.
• Data engineering should sit in contract talks to state the hardware mix, flexibility and exit terms.

Once the provider is chosen, the heaviest job on it is moving training data to the GPUs. → Training-Data Pipelines

Further reading

• CoreWeave documentation — CoreWeave, accessed 2026. Storage, networking and cluster options at a dedicated neocloud.
• Lambda pricing — Lambda, accessed 2026. Published GPU rates, useful as a reference point when comparing providers.
• RunPod pod pricing — RunPod, accessed 2026. How pods are billed, including the per-second model and storage charges.
• Vast.ai documentation — Vast.ai, accessed 2026. How the marketplace model works from the renter side.
• Amazon EC2 on-demand pricing — Amazon Web Services, accessed 2026. Includes the data transfer pricing that sets the egress side of a cross-cloud decision.