Training-Data Pipelines

Pipeline stages

A representative training-data pipeline for an LLM fine-tune or pre-train:

1. Ingest. Pull raw data from sources — web crawls, customer data, partner feeds, public datasets.
2. Filter and clean. Language detection, content quality scoring, PII redaction, dedup, license filtering.
3. Tokenize / pre-process. Convert to tokens (for LLM) or features (for other modalities).
4. Shard. Split into shards that parallel readers can consume.
5. Stage near GPU. Move to storage adjacent to the training cluster.
6. Register. Catalog the dataset version with provenance metadata.

Each stage has its own DE design decisions. Mistakes compound — bad sharding upstream means slow training downstream.

File formats

Common formats and their tradeoffs:

Parquet

The default for batch DE work. Columnar, well-compressed, broadly supported. Works for tokenized data when the workload reads selected columns.

WebDataset

Tar files containing record-per-file (image + label, etc.). Optimized for sequential streaming reads. Common in vision-and-multimodal work; useful when training jobs want to stream large media.

Mosaic Streaming (MDS)

A format designed for cloud-streaming during training. Includes shard manifests for resumable streaming. Common in modern LLM training stacks.

Arrow / Feather

In-memory columnar; useful for intermediate storage. Less common as final training-data format.

Raw JSONL or text

Common in research and early pipelines. Inefficient at scale; expensive to read; useful for inspection.

For LLM training, the realistic choice is between Parquet (warehouse-friendly, batched) and MDS or WebDataset (streaming-optimized). Pick based on whether you're feeding warehouse pipelines downstream or feeding training jobs directly.

Sharding

Sharding strategy determines how parallel training readers split the data:

• Shard size. Too small → metadata overhead and slow reads; too large → uneven work distribution. 100 MB to 1 GB is a common sweet spot.
• Number of shards. Should be a multiple of the number of parallel readers (typically GPU count × data loaders per GPU).
• Random ordering. Training expects to see varied data per batch. Either pre-shuffle the shards or randomize within them.
• Deterministic sharding. For reproducibility, the same dataset version should produce the same shards.

Sharding mistakes show up as GPU utilization gaps. If GPUs sit idle waiting for data, look at the shard size and reader parallelism first.

Moving data to GPUs

Three patterns for getting data into the GPU instance:

Pattern 1: Local SSD

Copy the dataset to the instance's local NVMe before training starts. Best throughput during training; longest startup time; fails if dataset exceeds local disk.

Use when dataset fits comfortably on local disk and the instance is dedicated (worth the staging cost).

Pattern 2: Network-attached storage

Mount a shared filesystem (provider-native or self-hosted) and read directly. No staging cost; ongoing network bandwidth cost; depends on storage throughput.

Use when storage and compute are co-located in the same cluster and the filesystem can deliver sufficient throughput.

Pattern 3: Streaming from object storage

Stream data from S3 (or equivalent) during training. Modern frameworks (Mosaic Streaming, WebDataset over HTTP, etc.) support this. No staging; tolerates resumption; depends on egress economics.

Use when the dataset is large enough that staging would be impractical and the provider has reasonable egress to your object store.

Mixed patterns work too — small shards local for speed, large shards streamed.

Deduplication & quality

The classic pre-train data engineering problems:

• Exact-string deduplication. Removes literal duplicates. MinHash + LSH is standard at scale.
• Near-duplicate detection. Same content with minor variation. More expensive; uses SimHash or learned embeddings.
• Quality scoring. Heuristic or model-based; filter out junk (gibberish, boilerplate, low-quality web content).
• Language detection. If training a specific-language model, filter accordingly.
• License compliance. Filter to licenses your training is allowed to use.
• PII removal. Names, emails, addresses scrubbed where required.

These stages can themselves run on neocloud compute when dataset scale exceeds warehouse practicality. Spark on a managed cluster or Ray-based pipelines on a GPU cloud are both viable.

Versioning

Every training run must reference a specific dataset version. Versioning approaches:

• Path-based. Datasets at s3://bucket/datasets/llm-train/2026-05-15/. Simplest; relies on convention.
• Content-addressed. Hash the shards; store under content hash. Stronger guarantees; harder to navigate.
• Catalog-tracked. A registry (Unity Catalog, MLflow, custom) tracks dataset versions with metadata.
• Lakehouse table versions. Iceberg / Delta tables versioned at the table level; training reads a specific snapshot.

For serious shops, catalog-tracked + content-addressed is the durable answer. Cheaper shops get by with path-based + convention until something breaks.

Streaming vs pre-materialized

Two philosophies for training data:

• Pre-materialized. The entire dataset is processed, sharded, and staged before training begins. Training reads a fixed corpus.
• Streaming. Training reads from a streaming source (live data or a streaming representation of static data). Useful when data continues to update or when staging is impractical.

Pre-materialized is more common for big training runs because it's reproducible. Streaming is more common for inference-side feedback loops and for very large pre-train datasets where staging the entire corpus is uneconomic.

Takeaway

Training-data pipelines are where DE work most directly meets GPU economics. Bad pipelines burn GPU hours; good pipelines keep GPUs saturated. The next chapter covers orchestration — how all this gets scheduled.