Storage for ML: Lakehouse, Formats & Dataset Versioning

Why storage is an ML problem, not a storage problem

Analytics storage is judged on one axis: how cheaply can a query engine scan it. ML storage is judged on three. The training loop needs throughput — a GPU node reading Parquet at 200 MB/s will sit idle while the accelerator waits for the next batch. The feature pipeline needs time travel — "the value of host_reliability_7d as of March 1st" is a storage question before it is a join question. And the model needs reproducibility — six months after a fraud model ships, a regulator or an incident review will ask what rows it trained on, and "roughly the March extract" is not an answer.

The good news is that the lakehouse stack most platforms already run — columnar files on object storage, governed by a table format — gets you two of the three for free. This chapter is about the third axis, the places where ML diverges from analytics (unstructured data, tensor-shaped records, random-access reads), and the discipline of versioning that turns a pile of Parquet into a dataset.

Table formats for ML: Parquet, Iceberg, Delta, Hudi

Start with the file layer. Parquet is the default for a reason: columnar, compressed, with per-row-group statistics so a reader can skip whole chunks. For tabular features — which is what ranking, fraud, churn, forecasting and pricing at GridDP all consume — nothing beats it for scan-heavy training reads. Its weaknesses show up at the edges: it is append-only at the file level (no in-place update), row-group granularity makes random access to a single record expensive, and it has no opinion about how many files make a table.

That last gap is what table formats fill. Iceberg, Delta Lake and Hudi all sit on top of Parquet and add a transaction log: which files belong to the table, what schema they carry, what snapshot they represent. For ML the three features that matter are snapshot isolation (a training job reads a consistent view while pipelines keep writing), time travel (query a past snapshot by ID or timestamp), and schema evolution (add a feature column without rewriting history).

The honest recommendation: pick the format your platform already governs with, and lean on its time travel for datasets. The mistake is not choosing "wrong" between these three; it is training from a table with no snapshot pinned at all, so that "the training set" silently changes every night the pipeline runs.

Tensors, images & text: when Parquet stops fitting

Tabular features are the easy case. The moment a model consumes images (a GPU benchmark screenshot classifier), long text (support-ticket embeddings), or pre-computed vectors, Parquet's row-group access pattern becomes a tax. A training loader wants to fetch row 4,183,022 and its 1,536-dimensional embedding in one seek; Parquet wants to hand you a 128 MB row group and let you find it.

Lance exists for exactly this. It is a columnar format with a fixed-size-list encoding for vectors, page-level random access, and a versioned dataset abstraction built in — you can append, add a column, and read any prior version without the table-format-on-top layer. LanceDB adds ANN indexes on the same files, which is why it shows up again in chapter 07. For multimodal training sets — image bytes alongside labels and metadata — the alternatives are WebDataset (tar shards streamed sequentially to GPU workers), TFRecord, or the "Parquet for metadata, object-store URIs for blobs" split.

Object-storage layout: feeding GPUs without starving them

S3, GCS and Azure Blob all have the same personality: high aggregate throughput, high per-request latency (tens of milliseconds), and a per-prefix request-rate ceiling. A training job that opens ten thousand 2 MB files will spend its life in request overhead; one that opens forty 512 MB files will saturate the network. The layout rules follow directly.

• File size. Target 128–512 MB per Parquet file for training reads; compact small files produced by streaming writers before they reach a dataset. Table formats ship compaction jobs — schedule them.
• Partitioning. Partition by what the training query filters on (usually event date and maybe entity type), never by high-cardinality keys. Over-partitioning recreates the small-file problem one directory at a time.
• Sort within files. Sort by the entity key used in point-in-time joins (account_id, host_id) so row-group statistics prune, and so a loader reading one entity's history touches few pages.
• Sharding for parallel loaders. A distributed training job with 8 workers wants at least 8×N roughly equal shards it can claim without coordination. WebDataset and Lance both make shard = file.
• Locality and egress. Keep the dataset in the same region as the GPUs. Cross-region egress at training scale costs more than the storage; for a neocloud renting GPUs in several regions, replicate the dataset once per active region rather than pulling it per job.

s3://griddp-lake/                      ← governed lakehouse (Iceberg catalog)
  silver/fct_telemetry_hourly/         ← mutable tables, snapshots expire in 7d
  silver/fct_rental/
  features/host_reliability_7d/        ← offline feature store, SCD2, snapshots kept 400d

s3://griddp-ml-datasets/               ← IMMUTABLE datasets (object lock / no delete role)
  fraud-signup/v2026.03.01-a1f9c/      ← one prefix per dataset version
    manifest.json                       ← snapshot ids, feature versions, row counts, splits
    train/part-000{0..63}.parquet       ← 64 shards × ~300 MB, sorted by account_id
    valid/part-000{0..07}.parquet
    test/part-000{0..07}.parquet
  ticket-embeddings/v2026.02.14-77bd0/
    data.lance                          ← vectors + metadata, versioned in-format

s3://griddp-ml-datasets-eu/            ← replica for EU GPU region, synced once per version

Dataset versioning: the immutable-artifact rule

Here is the rule, and it is not negotiable for anything that ships: a model trains from a named, immutable dataset version, and the model registry records that name. Not "the features table", not "last night's run" — a version with a manifest. There are four ways to get there, and they differ mostly in where the version lives.

For a platform team the pragmatic pattern is the first two combined: pin snapshots for the point-in-time join, then export the joined, split training set to an immutable prefix with a manifest. The export is what the trainer reads (fast, sharded, regional) and what the registry names; the snapshot IDs in the manifest are what let you regenerate it if the export is ever lost.

{
  "dataset": "fraud-signup",
  "version": "v2026.03.01-a1f9c",
  "created_at": "2026-03-01T04:12:09Z",
  "built_by": "pipelines/fraud_training_set@9e1d2c",
  "sources": {
    "features.acct_payment_age":   {"iceberg_snapshot": 6019873321, "registry_version": 4},
    "features.device_ip_velocity": {"iceberg_snapshot": 6019873402, "registry_version": 2},
    "labels.fraud_confirmed":      {"iceberg_snapshot": 6019870115, "label_delay_days": 45}
  },
  "point_in_time": {"prediction_ts_column": "signup_ts", "join": "asof"},
  "splits": {"train": {"rows": 4183022, "end": "2026-01-31"},
             "valid": {"rows":  512340, "start": "2026-02-01", "end": "2026-02-14"},
             "test":  {"rows":  498115, "start": "2026-02-15", "end": "2026-02-28"}},
  "schema_hash": "sha256:3c9f…",
  "immutable": true
}

Worked example: GridDP's four models, four storage shapes

Map the storage decisions onto the models from the case study:

• Ranking. Tabular features (DLPerf, reliability, price-vs-market) from Iceberg feature tables; nightly export of ~40 M offer-impression rows to a sharded Parquet dataset, sorted by offer_id. Snapshots tagged rank-train-2026-03-01 so the point-in-time join can be re-run.
• Fraud. Smaller (a few million signups) but the manifest matters most here — a regulator may ask. Exported dataset with the label-delay window recorded; Iceberg snapshots retained 400 days on the feature tables that feed it.
• Capacity forecasting. Trains on fct_telemetry_hourly — the firehose rolled up. This is the one place file layout dominates: partition by day, sort by (gpu_model, region, ts), compact to 256 MB files, and the forecaster reads two years of hourly rows in minutes instead of hours.
• Dynamic pricing. Bid-event sequences as nested lists in Parquet, one row per market-hour, so a sequence model gets contiguous input. Lance for the market-state embeddings the pricing team experiments with, versioned in-format.

Four models, one governed lakehouse, one immutable-dataset bucket, one manifest schema. The variety is in the file layout, not in the process.

When to choose what

The mature answer names the seam: "Features live in Iceberg with SCD2 history and long snapshot retention; a training set is a point-in-time join pinned to snapshot IDs, exported as sharded Parquet to an immutable bucket with a manifest, and the manifest's version string is what the model registry records." Everything in the next nine chapters assumes that sentence.

Next: the features that cannot wait for a nightly batch — 06 · Streaming Features & Real-Time Inference Data.

✓ Check yourself

Answer from memory before opening each one.

1. A model registry stores the table name and a date for each model's training data. Three weeks later the pipeline's maintenance job runs. What can go wrong, and what do you record instead?

   Show answer

   Snapshot expiry or vacuuming can delete the snapshot the date maps to, so the training set can no longer be rebuilt, and a table with no pinned snapshot silently changes with every pipeline run. Record a named immutable dataset version with a manifest of snapshot IDs, and tag or export what you train on. See table formats for ML.

2. A training job on 8 GPU workers reads ten thousand 2 MB Parquet files from object storage, and utilisation sits at 10%. Give the cause and the layout you would change it to.

   Show answer

   Object stores have high per-request latency, so small files spend the job in request overhead and the accelerators wait. Compact to roughly 128 to 512 MB files and write at least 8 times N similarly sized shards that workers can claim without coordination. See object-storage layout.

3. The loader needs row 4,183,022 and its 1,536-dimension embedding for each sample, at random. Why does Parquet make this expensive, and what do you pick instead?

   Show answer

   Parquet reads in row groups of around 128 MB, so one record costs a large read plus a search inside it. A format with page-level random access and a native vector encoding, such as Lance, fetches the record directly. See tensors, images and text.

4. Your fraud label arrives 45 days after signup, and the team splits at the end of the month to include every recent signup. What goes wrong, and where do you record the rule?

   Show answer

   The last 45 days of any window have no labels yet, so recent fraud is counted as non-fraud and validation understates fraud. Set split boundaries that respect the label delay and write them with the delay into the manifest, so no one guesses later. See the immutable-artifact rule.

5. A regulator may ask what the fraud model trained on. Which of pinned Iceberg snapshots, an immutable export or LakeFS do you rely on, and what decides it?

   Show answer

   Pin snapshots to rerun the point-in-time join, and also export the joined, split set to an immutable prefix with a manifest, because exports survive table churn, replicate cheaply to GPU regions and are easy to audit. Add LakeFS only if several teams need atomic multi-table versions and branches. See dataset versioning.

6. Compare what you would store for ranking features, ticket embeddings and benchmark screenshots, and say why they differ.

   Show answer

   Ranking features are tabular, so Parquet under the governed table format gives scan throughput and time travel. Embeddings are fixed-width vectors read at random, so Lance or Parquet with a fixed-size list and an external index. Images are blobs, so keep them in object storage, with a governed table of URIs and labels, and shard for the trainer. See tensors, images and text.

Takeaway

• ML storage is judged on throughput, time travel and reproducibility, and a table answers what is true now while a dataset answers what the model saw.
• A table format adds snapshot isolation, time travel and schema evolution to Parquet, so choose the one your platform already governs and pin snapshots instead of training from a moving table.
• Snapshot expiry deletes datasets, so tag training snapshots or export them to an immutable location before the first model ships.
• Vectors, images and sequences need different shapes: Lance or fixed-size lists for random access, blobs plus a manifest for media, and nested lists for sequences.
• Object stores reward few large shards in the training region and punish many small files, so compact, sort by the join key and replicate per version, not per job.
• A model trains from a named immutable dataset version with a manifest of snapshots, feature versions, splits and label delay, and the registry records that name.

Everything so far has been batch, so the next question is what to do with features that cannot wait for a nightly run. → Streaming Features & Real-Time Inference Data

Further reading

• Apache Iceberg table spec — Apache Iceberg, accessed 2026. Snapshots, manifests and metadata that make time travel and pinned reads possible.
• Branching and tagging — Apache Iceberg, accessed 2026. How to tag a snapshot so expiry does not remove a training set.
• Delta Lake batch reads and writes — Delta Lake documentation, accessed 2026. Time travel by version or timestamp, and the retention settings that limit it.
• WebDataset — WebDataset project, accessed 2026. Sharded tar format for streaming large training sets to many workers.
• lakeFS documentation — lakeFS, accessed 2026. Git-style branches and commits over object storage.