Storage & the Lakehouse

The OLTP↔OLAP divide

The source systems in chapter 01 are row-oriented: Postgres stores all the columns of a rental together, contiguously, so that fetching one rental by primary key is a single seek and a single read. That layout is perfect for the OLTP workload — point lookups, single-row mutations, transactional consistency — which is exactly what the marketplace product does thousands of times per second.

Analytics is the opposite workload. An analyst asks "average GPU utilization by data-center region last month" — a query that touches two columns across hundreds of millions of rows. In a row store, the engine must read every row in full (all 11 telemetry columns) just to extract the 2 it needs, then discard the rest. The disk and decompression work is dominated by columns the query never references. This is why you copy data out of Postgres into a column-oriented analytical store rather than pointing dashboards at the production database: the access patterns are fundamentally adversarial, and so are the optimal physical layouts.

Two free wins fall out of the columnar layout. First, projection pushdown: a query only reads the columns it names, so a 2-of-11-column query does ~18% of the I/O. Second, compression: a column is a run of same-typed, often-similar values (every temp_c near 70, every gpu_index in 0–7), which compresses far better than a heterogeneous row. GridDP's ~150–250 GB/day landed figure from Start Here is the compressed number precisely because we store columnar.

Columnar file formats: Parquet internals

Apache Parquet is the de-facto on-disk format for analytical data — it's what Iceberg, Delta, and Hudi all store underneath. Understanding its internal layout is what separates an engineer who uses a lakehouse from one who can tune it, because every partitioning and clustering decision later in this chapter is really a decision about what Parquet's metadata lets the engine skip.

The structure is hierarchical: a file holds row groups (horizontal slices of ~64–256 MB), each row group holds one column chunk per column, and each chunk holds pages of encoded values. The footer at the end carries the schema and, crucially, min/max/null-count statistics for every column chunk. Engines read the footer first, then decide what to read next.

Three mechanisms make analytical scans cheap:

• Projection pushdown — to read gpu_util, the engine seeks directly to that column chunk's byte offset (from the footer) and reads only those bytes. The other ten columns are never touched.
• Predicate pushdown via statistics — for WHERE temp_c > 90, the engine checks each row group's temp_c min/max. A row group with max=84 cannot contain a match, so it's skipped without decompressing a single value. On well-sorted data this prunes the vast majority of row groups. (Iceberg and Delta keep these same statistics at the file level too, so they can skip whole files before even opening them.)
• Encoding + compression — within a chunk, Parquet applies dictionary encoding (replace repeated values like machine_id with small integer codes), run-length encoding (RLE — store "97 repeated 40,000 times" once), and bit-packing, then a general codec (Snappy for speed, Zstd for ratio). Telemetry columns with low cardinality and physical locality routinely hit 10–20× compression.

Open table formats: from a pile of files to a real table

A directory full of Parquet files is not a table. There's no atomic commit (a reader can see a half-finished write), no way to roll back, no schema enforcement, no record of what files belong to which version. Open table formats — Apache Iceberg, Delta Lake, and Apache Hudi — add a metadata layer on top of the Parquet files that supplies exactly these database properties while leaving the data in open, engine-agnostic files on object storage.

The decoupling is the whole point: storage is cheap object-store bytes, the table format is a thin metadata layer, and any engine that speaks the format can read and write. That's the lakehouse promise — the cheap-storage / open-format virtues of a data lake plus the ACID / transactional virtues of a warehouse, without copying data into a proprietary engine. The four capabilities every format provides:

• ACID commits — a write either fully appears or doesn't; concurrent readers never see torn state (optimistic concurrency over a metadata log).
• Time travel — every commit is a snapshot; you can query the table "as of yesterday 14:00" or roll back a bad load. Invaluable for finance reconciliation and ML training-set reproducibility.
• Schema evolution — add, drop, rename, reorder columns by metadata change, without rewriting the data files.
• Partition evolution — change how the table is laid out going forward without rewriting history (an Iceberg specialty).

Mapping back to the three reference stacks from Start Here:

• Stack A (open-source lakehouse) picks Iceberg on S3. Rationale: GridDP wants multiple engines (Trino for ad-hoc SQL, Spark for heavy transforms, ClickHouse and DuckDB for specialized reads) hitting one copy of the data, with zero vendor lock-in. Iceberg's interop and partition evolution win decisively.
• Stack B (Databricks-centric) uses Delta Lake — it's native, deeply optimized in Photon, and pairs with Unity Catalog and the Feature Store. UniForm can expose Delta as Iceberg for outside readers when needed.
• Stack C (Snowflake-centric) uses Snowflake-native tables for the core warehouse, plus Iceberg tables for the open/large-scale data Snowflake reads in place — getting Snowflake ergonomics without locking every byte into FDN format.

Partitioning, hidden partitioning & clustering

Recall the lesson from Parquet: the engine skips data it can prove is irrelevant. Partitioning and clustering are how you arrange data so it can prove that for the queries people actually run. Get this right and a query reads gigabytes; get it wrong and the same query reads terabytes for the same answer.

Partitioning physically splits a table into directories by a column's value — event_date=2026-06-16/, event_date=2026-06-15/, and so on. A query with WHERE event_date = '2026-06-16' opens exactly one directory; the rest are pruned before any file is read. This is the single biggest lever for the telemetry table, where virtually every query is time-bounded.

The classic trap: with plain (Hive-style) partitioning, the partition column is a separate, derived column the writer must populate and the reader must filter on by its exact name. Query WHERE event_time > '2026-06-16' instead of the derived event_date and you get no pruning — a silent full scan. Iceberg's hidden partitioning fixes this: you declare a partition transform (days(event_time)) and Iceberg derives and tracks the partition value itself. Analysts filter on the natural event_time column and pruning just works — no leaky derived column, and the spec can evolve later (e.g. from daily to hourly) without rewriting old data.

CREATE TABLE fleet.telemetry_hourly (
    machine_id      STRING,
    gpu_index       INT,
    event_hour      TIMESTAMP,        -- truncated to the hour upstream
    avg_gpu_util    DOUBLE,
    max_temp_c      INT,
    avg_power_w     DOUBLE,
    p95_sm_clock    INT,
    sample_count    BIGINT,
    instance_id     STRING
)
USING iceberg
PARTITIONED BY (days(event_hour))      -- HIDDEN partition transform
TBLPROPERTIES (
    'write.target-file-size-bytes' = '268435456',   -- 256 MB
    'write.parquet.compression-codec' = 'zstd',
    'write.distribution-mode' = 'range',             -- sort on write
    'sort-order' = 'machine_id, event_hour'          -- cluster within partition
);

-- Later, with no rewrite of historical data, tighten to hourly partitions:
ALTER TABLE fleet.telemetry_hourly
  REPLACE PARTITION FIELD days(event_hour) WITH hours(event_hour);

Within a partition, clustering / sort order decides how tight each file's min/max stats are. GridDP's telemetry is queried two ways: "the whole fleet over a time window" (handled by the date partition) and "one machine over time" (handled by clustering on machine_id). Sorting files by machine_id, event_hour means a single-machine query touches only the few files whose machine_id range covers it — the rest are pruned by file-level stats.

The same idea wears different names across engines: Delta calls it Z-ordering (or now liquid clustering), Snowflake uses clustering keys with automatic reclustering, ClickHouse uses the sorting key (next section). All answer one question: given how this table is queried, how should rows be ordered on disk so the engine reads the least?

Object storage as the substrate — and the small-files problem

Underneath every lakehouse table sits object storage — S3, GCS, Azure Blob. It's the right substrate for analytics because it is effectively infinite, extremely durable (eleven nines), and cheap (cents per GB-month), and it decouples storage from compute so you pay for each independently. But it is not a filesystem, and the differences bite:

• Latency, not throughput, is the enemy. Each GET carries tens of milliseconds of overhead. Reading one 256 MB file is fast; reading 25,600 ten-KB files for the same bytes is dominated by per-request latency and will be orders of magnitude slower.
• No atomic rename. Object stores have no cheap "mv". This is precisely why table formats keep an explicit metadata log to make commits atomic, instead of relying on directory renames the way old Hive tables did.
• Eventually-consistent listings (historically). S3 is now strongly read-after-write consistent, but listing semantics and request throttling still make "just scan the directory" an anti-pattern at scale — another reason the format tracks the file list in metadata.

The dominant operational pain is the small-files problem, and the firehose causes it directly. Streaming ingest (chapter 04) commits frequently to keep data fresh — every micro-batch writes a new small Parquet file. At ~1,700 telemetry events/sec with minute-level commits, a single day's partition accretes thousands of tiny files. Query planning now has to open and stat all of them, footer reads dominate, and the metadata layer balloons.

The fix is a background compaction job (Iceberg: rewrite_data_files; Delta: OPTIMIZE / auto-compaction; Hudi: clustering) that periodically rewrites the many small files into a few right-sized (128–512 MB), sorted files — and the ACID layer makes the rewrite an atomic, invisible-to-readers commit. A companion job (Iceberg expire_snapshots + remove_orphan_files; Delta VACUUM) reclaims storage from old snapshots after the time-travel window. Compaction is not optional on a streaming table; it's a permanent, scheduled part of the platform, and chapter 14 budgets for the compute it consumes.

Storage tiering & lifecycle

Not all data is equal in value, and object stores price that directly. GridDP's ~150–250 GB/day → 50–90 TB/year growth means storage cost compounds, but the access pattern decays sharply: yesterday's telemetry is queried constantly; telemetry from 14 months ago is touched only for the occasional audit. Tiering matches storage class to access frequency.

Tiering is automated with lifecycle policies: a rule on the bucket transitions objects to a colder class after N days based on age. Because Iceberg/Delta tables are just objects on S3, the same policies apply — though you transition the data files while keeping recent metadata hot so planning stays fast. The downsampling pattern compounds the savings: raw 10-second telemetry need not live forever at full resolution. After 30 days, GridDP keeps hourly rollups (1/360 the rows) and lets the raw samples expire — covered concretely in the ClickHouse TTL below.

The specialist store for the firehose: ClickHouse

Everything above describes the lakehouse — the right home for business data and pre-aggregated rollups. But chapter 01's warning stands: do not route the raw firehose through the warehouse. The 150 M raw samples/day belong in a purpose-built time-series engine. Across all three reference stacks, GridDP runs ClickHouse for this — a columnar database engineered specifically for high-ingest, time-ordered, append-mostly data.

ClickHouse's storage engine is MergeTree. It's conceptually the same columnar+sorted idea as Parquet, but as a live, continuously-merging database: data is written in parts, each part stores columns sorted by the table's sorting key, and a background process merges parts (its own built-in answer to small files). The sorting key is the equivalent of clustering — it determines what ClickHouse can skip via its sparse primary index.

CREATE TABLE fleet.telemetry_raw (
    machine_id     LowCardinality(String),
    gpu_index      UInt8,
    event_time     DateTime64(3),
    gpu_util_pct   Float32,
    mem_used_mb    UInt32,
    temp_c         UInt8,
    power_w        UInt16,
    sm_clock_mhz   UInt16,
    ecc_errors     UInt32,
    instance_id    LowCardinality(String)
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(event_time)
ORDER BY (machine_id, gpu_index, event_time)     -- the sorting key
TTL event_time + INTERVAL 30 DAY                  -- expire raw samples
        GROUP BY machine_id, gpu_index, toStartOfHour(event_time)
        SET gpu_util_pct = avg(gpu_util_pct),     -- downsample in place
            temp_c       = max(temp_c),
            power_w      = avg(power_w);

-- a materialized view maintains hourly rollups on ingest, pushed to the lakehouse
CREATE MATERIALIZED VIEW fleet.telemetry_hourly_mv
TO fleet.telemetry_hourly AS
SELECT
    machine_id,
    gpu_index,
    toStartOfHour(event_time)            AS event_hour,
    avg(gpu_util_pct)                     AS avg_gpu_util,
    max(temp_c)                           AS max_temp_c,
    avg(power_w)                          AS avg_power_w,
    count()                               AS sample_count
FROM fleet.telemetry_raw
GROUP BY machine_id, gpu_index, event_hour;

Three ClickHouse features carry the firehose:

• Sorting key — ORDER BY (machine_id, gpu_index, event_time) matches the two query shapes (fleet-over-time and machine-over-time), so the sparse index prunes aggressively.
• TTL-based downsampling & expiry — the TTL … GROUP BY … SET clause rolls up data older than 30 days in place (averaging utilization, max-ing temperature) and the rest expires. Retention-per-resolution from the tiering section, expressed declaratively in the schema.
• Materialized views for rollups — incremental aggregation computed on insert, not at query time. The hourly rollup is maintained continuously and is the exact dataset that flows into the Iceberg telemetry_hourly table for joining against business data in the warehouse.

Why this is 10–100× cheaper than a warehouse for time-series: ClickHouse ingests millions of rows/sec on commodity hardware with no per-query compute billing, its sparse-index + sorting-key design is tuned for exactly the "metric over a time window for an entity" pattern, and TTL downsampling caps storage growth automatically. A general warehouse charges premium compute for every scan of data it stores row-expensively and never downsamples on its own.

Takeaway

• OLTP is row-oriented (point lookups, mutations); OLAP is column-oriented (scans, aggregations). You copy data out of Postgres into a columnar store because the access patterns demand opposite physical layouts.
• Parquet makes scans cheap via projection pushdown, predicate pushdown on min/max statistics, and dictionary/RLE/compression — and all of it only pays off when data is sorted on what you filter.
• Open table formats (Iceberg, Delta, Hudi) add ACID, time travel, and schema/partition evolution over Parquet files. GridDP picks Iceberg in Stack A (open, multi-engine), Delta in Stack B (Databricks-native), Snowflake-native + Iceberg in Stack C.
• Partitioning gives coarse pruning (by date); clustering/sort order gives the fine grain (by machine_id) without exploding partition counts. Iceberg's hidden partitioning removes the full-scan footgun.
• Object storage is the cheap, durable, infinite substrate — but its latency and lack of atomic rename make small files (from streaming) a real problem that compaction jobs must continuously fix.
• Tier by access frequency (hot/warm/cold/archive) and downsample by resolution; never keep full-fidelity telemetry forever.
• The firehose lives in ClickHouse, not the warehouse — MergeTree, sorting keys, TTL downsampling, and materialized-view rollups serve time-series 10–100× cheaper.

With the bytes laid down correctly, the next question is logical shape: how to model these landed files into facts, dimensions, and a canonical schema analysts can trust. → Data Modeling