Section B · DE using neoclouds

Training-Data Pipelines

The pipeline from raw source data to GPU-ready shards. Format choices, deduplication, sharding strategy, and the mechanics of moving terabytes of data into a training cluster without burning idle GPU hours.

Pipeline stages

A representative training-data pipeline for an LLM fine-tune or pre-train:

Ingest. Pull raw data from sources — web crawls, customer data, partner feeds, public datasets.
Filter and clean. Language detection, content quality scoring, PII redaction, dedup, license filtering.
Tokenize / pre-process. Convert to tokens (for LLM) or features (for other modalities).
Shard. Split into shards that parallel readers can consume.
Stage near GPU. Move to storage adjacent to the training cluster.
Register. Catalog the dataset version with provenance metadata.

Each stage has its own DE design decisions. Mistakes compound — bad sharding upstream means slow training downstream.

File formats

Common formats and their tradeoffs:

Parquet

The default for batch DE work. Columnar, well-compressed, broadly supported. Works for tokenized data when the workload reads selected columns.

WebDataset

Tar files containing record-per-file (image + label, etc.). Optimized for sequential streaming reads. Common in vision-and-multimodal work; useful when training jobs want to stream large media.

Mosaic Streaming (MDS)

A format designed for cloud-streaming during training. Includes shard manifests for resumable streaming. Common in modern LLM training stacks.

Arrow / Feather

In-memory columnar; useful for intermediate storage. Less common as final training-data format.

Raw JSONL or text

Common in research and early pipelines. Inefficient at scale; expensive to read; useful for inspection.

For LLM training, the realistic choice is between Parquet (warehouse-friendly, batched) and MDS or WebDataset (streaming-optimized). Pick based on whether you're feeding warehouse pipelines downstream or feeding training jobs directly.

Sharding

Sharding strategy determines how parallel training readers split the data:

Shard size. Too small → metadata overhead and slow reads; too large → uneven work distribution. 100 MB to 1 GB is a common sweet spot.
Number of shards. Should be a multiple of the number of parallel readers (typically GPU count × data loaders per GPU).
Random ordering. Training expects to see varied data per batch. Either pre-shuffle the shards or randomize within them.
Deterministic sharding. For reproducibility, the same dataset version should produce the same shards.

Sharding mistakes show up as GPU utilization gaps. If GPUs sit idle waiting for data, look at the shard size and reader parallelism first.

Moving data to GPUs

Three patterns for getting data into the GPU instance:

Pattern 1: Local SSD

Copy the dataset to the instance's local NVMe before training starts. Best throughput during training; longest startup time; fails if dataset exceeds local disk.

Use when dataset fits comfortably on local disk and the instance is dedicated (worth the staging cost).

Pattern 2: Network-attached storage

Mount a shared filesystem (provider-native or self-hosted) and read directly. No staging cost; ongoing network bandwidth cost; depends on storage throughput.

Use when storage and compute are co-located in the same cluster and the filesystem can deliver sufficient throughput.

Pattern 3: Streaming from object storage

Stream data from S3 (or equivalent) during training. Modern frameworks (Mosaic Streaming, WebDataset over HTTP, etc.) support this. No staging; tolerates resumption; depends on egress economics.

Use when the dataset is large enough that staging would be impractical and the provider has reasonable egress to your object store.

Mixed patterns work too — small shards local for speed, large shards streamed.

Deduplication & quality

The classic pre-train data engineering problems:

Exact-string deduplication. Removes literal duplicates. MinHash + LSH is standard at scale.
Near-duplicate detection. Same content with minor variation. More expensive; uses SimHash or learned embeddings.
Quality scoring. Heuristic or model-based; filter out junk (gibberish, boilerplate, low-quality web content).
Language detection. If training a specific-language model, filter accordingly.
License compliance. Filter to licenses your training is allowed to use.
PII removal. Names, emails, addresses scrubbed where required.

These stages can themselves run on neocloud compute when dataset scale exceeds warehouse practicality. Spark on a managed cluster or Ray-based pipelines on a GPU cloud are both viable.

Versioning

Every training run must reference a specific dataset version. Versioning approaches:

Path-based. Datasets at s3://bucket/datasets/llm-train/2026-05-15/. Simplest; relies on convention.
Content-addressed. Hash the shards; store under content hash. Stronger guarantees; harder to navigate.
Catalog-tracked. A registry (Unity Catalog, MLflow, custom) tracks dataset versions with metadata.
Lakehouse table versions. Iceberg / Delta tables versioned at the table level; training reads a specific snapshot.

For serious shops, catalog-tracked + content-addressed is the durable answer. Cheaper shops get by with path-based + convention until something breaks.

Streaming vs pre-materialized

Two philosophies for training data:

Pre-materialized. The entire dataset is processed, sharded, and staged before training begins. Training reads a fixed corpus.
Streaming. Training reads from a streaming source (live data or a streaming representation of static data). Useful when data continues to update or when staging is impractical.

Pre-materialized is more common for big training runs because it's reproducible. Streaming is more common for inference-side feedback loops and for very large pre-train datasets where staging the entire corpus is uneconomic.

✓ Check yourself

Answer from memory before opening each one.

GPU utilization sits near 55 percent and the data loaders are waiting on I/O. The corpus is 2,000 shards of about 5 MB each. What do you check first, and what do you change?

Show answer
Shard size. Thousands of tiny shards add request and metadata overhead to every read, which starves the GPUs. Re-shard toward the 100 MB to 1 GB range, then check reader parallelism. See sharding.
A run uses 48 GPUs with 8 data loaders each, and the dataset has 1,000 shards. What goes wrong, and what shard count would you pick?

Show answer
There are 384 readers, and 1,000 is not a multiple of 384, so some readers get three shards and some get two. The faster readers finish early and sit idle while the rest catch up. Pick a multiple of 384 such as 1,152, which gives each reader three. See sharding.
One corpus feeds warehouse analytics and also feeds training jobs directly. Parquet or a streaming format such as MDS or WebDataset?

Show answer
Judgement call. Parquet fits warehouse tools and batched work, while MDS and WebDataset fit sequential streaming into the loaders. A common answer is Parquet as the source of record and derived streaming shards for training. Choose by who reads it first. See file formats.
A 12 TB dataset must reach a dedicated cluster for ten days. Each node has 2 TB of local NVMe. Which of the three movement patterns can you rule out, and what decides between the rest?

Show answer
Local SSD staging is out, because the dataset does not fit on a node. Network-attached storage and streaming from object storage remain. Storage throughput and co-location favor the shared filesystem, while egress cost and the lack of a staging step favor streaming. See moving data to GPUs.
Why does path-based dataset versioning fail first, and what do content addressing and a catalog add?

Show answer
It rests on convention, so someone overwrites a path and an old run can no longer be reproduced. Content hashes make a change detectable, and a catalog records provenance and which run used which version. Together they make reproduction checkable. See versioning.
When is streaming training data the right choice over pre-materialized, given that pre-materialized is the reproducible default?

Show answer
When the corpus is too large to stage economically, or when data keeps updating, as in feedback loops from inference. You trade reproducibility for feasibility, so record what was read. See streaming vs pre-materialized.

Takeaway

A training-data pipeline runs ingest, filter, tokenize, shard, stage near the GPUs and register, and an early mistake compounds into slow training later.
Format follows the reader: Parquet when warehouses consume the data, MDS or WebDataset when training streams it.
Shards of roughly 100 MB to 1 GB, in a count that divides evenly across readers, keep GPUs fed; idle GPUs point at shard size and reader parallelism first.
Local SSD, shared filesystem and object-store streaming trade startup time, bandwidth cost and egress; choose by dataset size and co-location.
Every run must name an exact dataset version, and catalog-tracked plus content-addressed versioning is the durable answer.
Pre-materialized data is the default for reproducibility; streaming is for corpora too large to stage.

Those pipelines have to be scheduled, retried and torn down across providers. → Orchestration Across Providers