Section B · DS using neoclouds

Cost-Aware ML Experimentation

The DS-specific tradeoffs when compute is the dominant cost. When spot makes sense, how to allocate budget across experiments, the value of iterating at smaller scale, and the leverage points where small DS decisions translate to large compute savings.

Why cost-awareness matters for DS

In a 2026 AI organization, the DS function's compute spend often exceeds salary spend. Wasteful experiments — runs that should have been smaller, sweeps that should have been narrower, baselines that should have been cheaper — directly hit the company's economics.

Cost-aware DS isn't penny-pinching; it's recognizing that the same insight can often be obtained at fractional cost with better experimental design.

When to use spot

Spot / interruptible / preemptible instances offer 50%+ discount in exchange for being killable. ML uses:

Use: Hyperparameter sweeps where each trial is small.
Use: Long training runs with frequent checkpointing.
Use: Evaluation suites that run overnight.
Use: Distillation / batch inference.
Don't use: Time-critical training (blocking a release).
Don't use: Training where checkpointing is expensive or unreliable.
Don't use: Interactive / debugging sessions.

Allocating budget across experiments

With a finite compute budget and many possible experiments, prioritization matters:

Highest-impact first. Experiments most likely to inform a production decision get full-scale resources.
Quick wins second. Easy experiments that resolve open questions cheaply.
Long-shot last. Speculative experiments at minimal scale unless preliminary results justify scaling.
Always-on: Critical eval suites and production-monitoring runs.

Some teams formalize this with explicit budget allocations per project. Others rely on DS judgment. Either works; both benefit from cost visibility (the DE-side cost engineering supports this).

Early stopping

Early stopping in hyperparameter sweeps:

Successive halving: Run many trials briefly, keep the top fraction, run them longer, repeat.
Hyperband: Variant of successive halving that allocates budget more efficiently.
Population-based training: Trials evolve hyperparameters during training based on relative performance.
Manual checkpointing: Stop trials that aren't promising based on the run's own metrics.

Even imperfect early stopping cuts sweep cost by 5-10x. The DS investment is worth it.

Iteration at smaller scale

A subtle but high-leverage DS practice: test ideas at smaller scale before committing to large runs.

Test architecture changes on a small subset of data and a small model first.
Validate training loop correctness with a small overfitting run.
Catch obvious bugs without spending GPU-days.
Decide which big experiments to actually run.

This discipline trades a small amount of upfront iteration time for major cost savings on doomed full-scale runs.

Leverage points

Specific DS decisions that have outsized cost impact:

Batch size. Bigger batches use GPU more efficiently up to a point; over that point, returns diminish and convergence sometimes suffers.
Mixed precision (fp16 / bf16 / fp8). Often free quality; ~2x compute throughput.
Gradient checkpointing. Trades compute for memory; lets you fit bigger models on smaller GPUs.
Data efficiency. Better data quality often beats more data; lower compute.
Sample efficiency in fine-tuning. Fewer high-quality examples often beats more low-quality.
Right-sized models. A 7B model that's good enough is cheaper than a 70B model that's slightly better.

✓ Check yourself

Answer from memory before opening each one.

A spot instance costs half the on-demand price. The job checkpoints every 2 hours, is preempted on average once every 6 hours of wall time and needs 15 minutes to restart. Roughly how much longer does it run, and is spot still cheaper?

Show answer
Each interruption loses about half the checkpoint interval (1 hour) plus the 15-minute restart, so about 1.25 hours is wasted in each 6 hours. Useful time is 4.75 of 6 hours, so the run takes about 26% longer and costs about 63% of on-demand. Spot still wins, but the margin shrinks as checkpoints get sparser. See when to use spot.
A fine-tune blocks a release due on Friday. Do you run it on spot?

Show answer
No. A preemption can push the finish past the deadline, and the saving does not matter if the release slips. Time-critical work, runs with expensive or unreliable checkpointing and interactive debugging stay on stable capacity. See when to use spot.
Your colleague wants to move from a 7B to a 70B model for a two-point metric gain. What do you ask before agreeing?

Show answer
Ask what two points are worth to the product, whether better data or a smaller gain from tuning would reach the same level and whether the 7B is already good enough. A smaller model that is good enough is cheaper to train and to serve. See leverage points.
Before launching a large run you train on a tiny subset until it overfits. What should you see, and what does it mean if you do not?

Show answer
The loss should fall close to zero. If it does not, the bug is in the loop, the data, the labels or the learning rate, and a full-scale run would burn GPU-days on it. See iteration at smaller scale.
A speculative idea shows a small improvement in a run that used 1% of the budget. Do you scale it up?

Show answer
It depends on how noisy the small-scale signal is, whether the effect usually survives scaling, the cost of the full run and the value if it holds. If the signal is above noise and the full run is affordable, scale in steps rather than all at once. See allocating budget.
What does early stopping in a sweep assume, and where does it mislead?

Show answer
It assumes an early metric ranks trials the way the final metric would. It misleads on trials that start slowly and finish well, and on noisy metrics. Even imperfect stopping saves a lot, so accept some regret. See early stopping.

Takeaway

For a DS team, compute spend can exceed salary spend, so wasted experiments hit the company's economics directly.
Spot suits small sweep trials, frequently checkpointed training, overnight evaluation and batch inference, and not deadline work or debugging.
Spend budget on experiments most likely to inform a production decision, keep long shots small until early results justify scaling.
Successive halving, Hyperband and population-based training stop weak trials early and can cut sweep cost several times over.
Small overfitting runs and small-model tests catch bugs before they cost GPU-days.
Batch size, mixed precision, data quality and right-sized models are the decisions with the largest cost impact.

The next chapter moves to the DS work inside a neocloud, starting with how it sets prices. → Pricing Models