Cost, Scale & Feeding the GPUs

Where the money goes

Ask a team where their ML data budget goes and they will say "storage". Measure it and the answer is usually different. For a platform at GridDP's scale the cost stack, roughly in descending order, is: GPU hours lost to input starvation (the training job runs but the accelerator sits at 30% utilization because the dataloader can't keep up); feature computation (recomputing large aggregations on a schedule regardless of whether anything changed); warehouse compute for training-set builds (point-in-time joins over years of history, run per experiment); egress (moving datasets between the lake's region and wherever the GPUs are); and only then storage, which is cheap per byte but compounds through copies — bronze, silver, feature history, per-experiment training snapshots, served-feature logs.

That ordering dictates the priorities. The first engineering hour goes to the loading path, not to Parquet compression settings.

Storage tiers & the math

Storage is cheap but not free, and the lever is tiering by access pattern, not compression heroics. Four tiers cover an ML data plane:

The math for GridDP's dominant source, the telemetry firehose: ~150M samples/day at ~120 bytes compressed ≈ 18 GB/day ≈ 6.5 TB/year in bronze. Keep 90 days hot-ish in the lake (1.6 TB), the hourly rollup forever (~2% of raw), and tier the rest. Feature history for the ~40 tier-1 features across 17,000 GPUs and 200k accounts at hourly grain is another ~1 TB/year. The copies are the surprise: if every experiment materializes its own training set and nobody expires them, a team of eight data scientists produces several TB a quarter. The fix is snapshot-based training sets (chapter 05) — a training set is a query plus a snapshot ID, not a copy — with a 90-day expiry on any physical materialization not referenced by a registered model.

The training data-loading path

The path from object store to GPU memory is where ML data architecture differs most from analytics. A warehouse query returns a result set; a training job needs a continuous, shuffled, sharded stream of samples delivered faster than the model consumes them, resumable mid-epoch, across many workers. Tabular models at GridDP scale (tens of millions of rows) are easy — the whole set fits on one trainer's NVMe. The firehose-derived forecasting model and the ticket-text models are not, and any multimodal or embedding work is decidedly not.

The proven shape is sharded sequential files plus a streaming dataset library:

• Shard the dataset into files of 64–512 MB — Parquet row groups for tabular, WebDataset tar shards for images/audio/text, or the library's native format (MosaicML Streaming's MDS, Ray Data blocks). Sequential reads of large objects are what object storage is good at; millions of small files are what it is terrible at.
• Pre-shuffle at write time so that reading shards in order is already random enough; then shuffle within a buffer at read time. A global shuffle at read time over object storage is a random-I/O disaster.
• Stream, don't stage — libraries like MosaicML Streaming, WebDataset, or Ray Data read shards directly from S3/GCS with prefetch and a local cache, so a 5 TB dataset trains on a node with 1 TB of disk and the first step starts in seconds rather than after a copy.
• Deterministic resumption — the loader records shard index and offset per worker so a preempted job (interruptible GPUs are GridDP's own product) resumes mid-epoch without replaying or skipping samples.

from streaming import StreamingDataset          # MosaicML Streaming
from torch.utils.data import DataLoader

# Shards were written pre-shuffled, 256 MB each, to s3://griddp-ml/datasets/ticket_intent/v7/
dataset = StreamingDataset(
    remote="s3://griddp-ml/datasets/ticket_intent/v7/",
    local="/nvme/cache/ticket_intent_v7",   # local shard cache, evicted LRU
    shuffle=True, shuffle_seed=42,           # deterministic across restarts
    batch_size=256,
    predownload=8 * 256,                     # keep 8 batches ahead of the GPU
    cache_limit="400gb",
)
loader = DataLoader(dataset, batch_size=256, num_workers=8, pin_memory=True,
                    prefetch_factor=4, persistent_workers=True)

# On preemption: StreamingDataset.state_dict() captures (epoch, sample offset) per rank;
# the job checkpoints it with the model and resumes exactly where it stopped.

Throughput to the GPU

The loader has to deliver bytes faster than the model eats them; everything else is secondary. Work the numbers before the first run:

1. Samples per second the GPU can process — from a short profiling run with synthetic data: say 3,000 samples/s for the ticket-text model on one H100.
2. Bytes per sample after decoding: 2 KB tokenized → 6 MB/s per GPU. Trivial. For an image model at 150 KB/sample and 800 samples/s it is 120 MB/s per GPU — with 8 GPUs, ~1 GB/s per node, which is where object-store throughput, NIC bandwidth, and decode CPU all start to matter.
3. CPU decode budget — JPEG decode and augmentation are usually the real bottleneck; count cores per GPU (8–16 is typical) or move decode to the GPU (NVIDIA DALI).
4. Object-store parallelism — a single S3 stream tops out around 50–100 MB/s; you reach 1 GB/s with 16–32 concurrent shard reads, which is what prefetch and multiple workers give you.

Then measure the only metric that matters: GPU utilization during training (and its cousin, dataloader wait time per step). Below 80% sustained, the money is going into idle silicon and the fix is in the data path, not the model. GridDP sells GPU-hours; it should be the last company to waste them.

Feature-computation cost

Feature pipelines are the quiet budget line: they run on a schedule whether or not anything changed, and windowed aggregations over long histories are expensive to recompute from scratch. Three levers:

• Incremental, not full recompute. host_reliability_7d should update from the last hour's events plus a sliding-window subtraction, not rescan seven days of telemetry every hour. dbt incremental models with a lookback window, or Flink windowed aggregates, make this the default.
• Materialize by demand. A feature computed for 200k accounts hourly, of which 4k are scored per day, is 98% waste. Compute batch features for the active population; compute on-demand features at request time from the online store's raw inputs.
• Stream when freshness pays for it. Streaming features (chapter 06) cost a permanently running job; batch costs a burst. bid_depth_5m for pricing needs the stream; acct_spend_trend_30d for churn does not. Put each feature on the cheapest path that meets its freshness SLA.

Tag every feature pipeline with its cost per run and its consumers. A feature with no consuming model in the registry for 60 days gets its pipeline paused automatically — the registry knows, so let it act.

Egress & locality

Egress is the cost line that teams discover on the invoice. Cloud providers charge ~$0.05–0.09 per GB to move data out of a region; a 5 TB dataset pulled to GPUs in another cloud costs $250–450 per training run, and teams run many. Two rules keep it bounded:

• Put the training data where the GPUs are, once. Replicate the sharded dataset to a bucket in the GPU region (or in the neocloud's object storage) and pin the trainer's cache there. Pay egress once per dataset version, not per experiment.
• Move the feature store's online tier to the inference region, and keep the offline store next to the warehouse. Cross-region reads on the hot path are both slow and metered.

This is the natural bridge to the sibling guide Data Engineering for Neoclouds: the cheapest GPU-hour is worthless if the data lives three regions away.

Cost governance

Governance is not a spreadsheet; it is attribution built into the plane. Tag every dataset, feature pipeline, training job, and online-store namespace with an owner and a model, export cost by tag into the same gold marts that hold everything else, and show it back on a dashboard per team and per model. Then three policies with teeth:

• Budgets per model tier. Tier-1 models (ranking, fraud, pricing) get a budget and an on-call; experiments get a per-scientist monthly cap with a soft warning at 80%.
• Expiry as the default. Training materializations, experiment artifacts, and unpinned snapshots expire on a clock unless referenced by a registered model.
• Utilization gates. A training job whose GPU utilization stays below 60% for ten minutes is flagged; repeat offenders are blocked from the shared queue until the loader is fixed.

The reporting side reuses the semantic layer: ml_cost_per_model and gpu_idle_hours are metrics defined once, beside gmv, in the same governed layer (DPE guide, chapter 08).

GridDP by the numbers

A plausible monthly cost profile for GridDP's ML data plane, at 17,000 GPUs on the fleet and five production models, after the fixes in this chapter:

Storage was the smallest line before and after. The thousand GPU-hours were the story — and at neocloud prices that line alone is the salary of the engineer who fixed it.

When to choose what

Next, the three ways the whole plane can be assembled, and how to pick one — 13 · Reference Architectures Compared.

✓ Check yourself

Answer from memory before opening each one.

1. A training job runs at 30% GPU utilisation and the team proposes better Parquet compression to cut cost. Why is this the wrong first move, and what do you measure?

   Show answer

   Storage is usually the smallest line, and idle GPU hours dominate. Measure GPU utilisation and dataloader wait time per step, and if utilisation sits below about 80%, fix the data path before anything else. See where the money goes.

2. An image model needs 800 samples per second at 150 KB per sample on 8 GPUs. What throughput must the node sustain, and which limits does it reach first?

   Show answer

   About 120 MB/s per GPU, so around 1 GB/s per node. A single object-store stream gives roughly 50 to 100 MB/s, so you need 16 to 32 concurrent shard reads, and CPU decode of JPEGs and augmentation is often the real limit, at 8 to 16 cores per GPU. See throughput to the GPU.

3. Why is a global shuffle at read time over object storage a problem, and what do you do at write time?

   Show answer

   It turns sequential reads of large objects into random I/O across many small ranges, which object stores handle badly. Pre-shuffle when writing shards, then shuffle within a buffer at read time. See the training data-loading path.

4. A spot GPU is preempted mid-epoch. What state must the loader have saved for a resume that neither replays nor skips samples?

   Show answer

   The epoch, shard index and sample offset per worker or rank, restored from the loader's state and checkpointed with the model, plus the same shuffle seed. See the training data-loading path.

5. A feature is computed hourly for 200,000 accounts, and 4,000 are scored per day. What do you change, and which other levers cut feature cost?

   Show answer

   Compute batch features only for the active population and the rest on demand, which removes about 98% of the work. Also update windows incrementally instead of rescanning, and put each feature on the cheapest path that meets its freshness SLA, pausing pipelines that have no consumer. See feature-computation cost.

6. A scientist pulls a 5 TB dataset to GPUs in another cloud for every experiment. Estimate the egress cost per run and give the design that fixes it.

   Show answer

   At $0.05 to $0.09 per GB, about $250 to $450 a run. Replicate the sharded dataset to the GPU region once per dataset version and pin the trainer's cache there, so egress is paid once, not per experiment. See egress and locality.

Takeaway

• The expensive resource in an ML data plane is a GPU waiting for data, so price every decision in GPU-hour equivalents and measure utilisation first.
• A terabyte-month of warm object storage costs about as much as seven hours of one H100, so tiering matters less than keeping the accelerator busy.
• Feed training from pre-shuffled, sharded files of 64 to 512 MB through a streaming loader with prefetch and resumable state, not from a warehouse query per run.
• Do the throughput arithmetic before the first run: samples per second times bytes per sample, then check object-store parallelism and decode CPU.
• Feature pipelines run whether or not anything changed, so compute incrementally, only for active entities, and stream only where freshness pays.
• Pay egress once per dataset version, tag cost to owners and models, expire by default and gate jobs on utilisation.

With the costs priced, the remaining choice is how to assemble the whole plane. → Reference Architectures Compared

Further reading

• MosaicML Streaming documentation — Databricks Mosaic, accessed 2026. Sharded, resumable streaming datasets from object storage for multi-node training.
• Ray Data — Ray documentation, accessed 2026. Streaming execution for data loading and preprocessing for training.
• Best practices design patterns: optimizing Amazon S3 performance — AWS documentation, accessed 2026. Request parallelism and per-prefix limits that set object-store throughput.
• Amazon S3 pricing — AWS, accessed 2026. Storage tiers, retrieval and data-transfer charges behind the cost arithmetic.
• NVIDIA DALI documentation — NVIDIA, accessed 2026. Moving image decode and augmentation from CPU to GPU.