Files & Serialization at Scale

The format is a decision, not an afterthought

Beginners reach for CSV by reflex — it opens in a spreadsheet, it's human-readable, every tool accepts it. For small files that's fine. But the moment your data grows, the file format becomes a performance and cost decision, and CSV is usually the wrong one.

Here's the concrete stake. In Chapter 01 you saw that a file is just serialized bytes. In Chapter 03 you met the distinction between row-oriented and columnar storage inside a database. File formats make exactly the same choice — but now you can see and touch the result, because it's a file on disk you can measure. A dashboard that scans a 50 GB CSV every time someone clicks a filter will be slow and run up a cloud bill; the identical data in partitioned, compressed Parquet might scan 2 GB and cost a tenth as much. Nothing about the data changed — only its physical layout.

Row formats vs columnar formats

Every file format has to decide one thing: as it lays bytes down on disk, does it keep a whole record together, or a whole column together? That single choice splits the format world in two.

• Row-oriented: CSV, JSON (lines), Avro. Values for one record sit next to each other.
• Columnar: Parquet, ORC. Values for one column sit next to each other.

Picture a tiny table of three rows and three columns, and look at the byte order each layout writes:

Why does columnar make files smaller? Because similar values end up adjacent. A column of region values is just us, eu, us, us, eu… — lots of repetition, which compresses brilliantly. A column of timestamps increases steadily, so you can store the differences instead of full values. When you mix all three columns together (the row layout), the byte stream looks random and compresses poorly.

Why does columnar make analytics faster? Analytics queries are wide and shallow: SELECT region, SUM(amount) … GROUP BY region touches 2 of maybe 200 columns, but every row. In a columnar file the engine reads only the region and amount chunks and skips the other 198 entirely. In a row file it must read every byte of every row to get at those two fields.

Inside a Parquet file

Parquet is the default analytics format you'll meet everywhere — DuckDB, Spark, BigQuery, Snowflake all read and write it. It's worth understanding its insides, because they explain why a well-targeted query barely touches the disk.

A Parquet file is a nested structure, not a flat dump:

Three things in there change the game:

• Embedded schema and types. Unlike CSV — where everything is text and "2026-06-21" might be a date or a string, you can't tell — Parquet stores the real type of each column. An engine reads a Parquet file already knowing column 3 is a 64-bit integer. No guessing, no parsing dates out of strings.
• Column chunks. Because each column lives in its own chunk, an engine seeks straight to the chunks it needs and reads nothing else. This is the file-level version of the columnar advantage above.
• Per-column statistics. The footer records the min and max value of each column within each row group. This is the quiet superpower.

Those statistics enable predicate pushdown (also called row-group skipping). Suppose a row group's date column has min = 2026-01-01 and max = 2026-01-31, and your query says WHERE date = '2026-06-21'. The engine reads just the footer, sees that June 21 cannot possibly be in a January-only row group, and skips the entire row group without reading it. On a file with hundreds of row groups, a selective filter might touch only one or two — that's how "WHERE date = X" stays cheap even on huge files.

Compression: the CPU-vs-size tradeoff

On top of the columnar layout, Parquet compresses each page with a codec. Compression shrinks the bytes on disk (less storage cost, fewer bytes to read from disk or network) at the price of CPU time to compress when writing and decompress when reading. That's the whole tradeoff: smaller files cost more CPU. The art is picking where to sit on that curve.

Columnar data compresses far better than row data for the reason we saw: a column is full of similar, often repeated values, and compression algorithms feed on repetition. The same gzip applied to a CSV might cut it in half; applied to columnar pages it can cut it by 5×.

Partitioning and the small-files problem

Row-group skipping happens inside a file. Partitioning applies the same idea one level up — across files. You split a dataset into a folder tree keyed by a column, most often a date:

Now a query filtered on date never even opens the other days' files — the engine prunes them by reading the directory names. This is wildly effective for the most common analytics filter (a date range), which is why date partitioning is the default move.

But there's a trap, and it's the one beginners fall into: the small-files problem. If you partition too finely — say by minute, or by date and region and user — you end up with millions of tiny files. Every file carries fixed overhead (open it, read its footer, list it in the directory), so thousands of 10 KB files are dramatically slower to scan than a handful of 256 MB files holding the same rows. Object storage especially punishes many small reads.

The fix when small files accumulate is compaction: a periodic job that reads many small files in a partition and rewrites them as a few large ones.

Object storage: the substrate of the lake

Where do all these Parquet files live? Increasingly, in object storage — Amazon S3, Google Cloud Storage, or the MinIO server you stood up locally in Course 2. Object storage is the substrate that makes the modern data platform possible:

• Cheap. Far cheaper per gigabyte than database storage or attached disks.
• Durable. Providers replicate your bytes across machines and facilities; data loss is vanishingly rare.
• Effectively infinite. You never run out of space or manage volumes — you just write more objects.

The catch: object storage has higher latency than a local disk. Each object is fetched over the network via an HTTP request, so a single read is slow to start, even though throughput once flowing is high. This is precisely why the techniques above matter so much here — column chunks, row-group skipping, and partition pruning all minimize the number and size of reads, which is exactly what high-latency object storage rewards.

Hands-on: measure CSV vs Parquet yourself

Claims about "5–10× smaller and faster" only stick once you've seen them. Let's generate a dataset, write it both ways, and compare. We'll use DuckDB from Python (installed in Course 2; if not, pip install duckdb). DuckDB is itself a columnar engine and reads/writes Parquet natively.

import duckdb, os, time

con = duckdb.connect()

# 1) Build a realistic-ish table: 5 million rows, a few repetitive columns.
con.execute("""
    CREATE TABLE events AS
    SELECT
        i                                   AS id,
        DATE '2026-01-01' + (i % 180)       AS event_date,   -- 180 distinct days
        ['us','eu','apac'][1 + (i % 3)]     AS region,        -- only 3 values
        (i * 7 % 1000) / 10.0               AS amount
    FROM range(5_000_000) t(i);
""")

# 2) Write the SAME data two ways.
con.execute("COPY events TO 'events.csv'     (FORMAT CSV, HEADER);")
con.execute("COPY events TO 'events.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);")

# 3) Compare file sizes.
csv_mb = os.path.getsize('events.csv') / 1e6
pq_mb  = os.path.getsize('events.parquet') / 1e6
print(f"CSV    : {csv_mb:8.1f} MB")
print(f"Parquet: {pq_mb:8.1f} MB   ({csv_mb / pq_mb:.1f}x smaller)")

# 4) Compare scan speed for a typical analytics query.
def timed(sql):
    start = time.perf_counter()
    con.execute(sql).fetchall()
    return time.perf_counter() - start

q = "SELECT region, SUM(amount) FROM read_{}('events.{}') WHERE event_date = DATE '2026-03-01' GROUP BY region"
t_csv = timed(q.format('csv', 'csv'))
t_pq  = timed(q.format('parquet', 'parquet'))
print(f"\nScan CSV    : {t_csv*1000:7.1f} ms")
print(f"Scan Parquet: {t_pq*1000:7.1f} ms   ({t_csv / t_pq:.1f}x faster)")

Prefer pandas? The same comparison, fewer moving parts:

import pandas as pd, numpy as np, os

n = 5_000_000
df = pd.DataFrame({
    "id": np.arange(n),
    "event_date": pd.Timestamp("2026-01-01") + pd.to_timedelta(np.arange(n) % 180, "D"),
    "region": np.array(["us", "eu", "apac"])[np.arange(n) % 3],
    "amount": (np.arange(n) * 7 % 1000) / 10.0,
})

df.to_csv("events.csv", index=False)
df.to_parquet("events.parquet", compression="zstd")   # needs pyarrow

csv_mb = os.path.getsize("events.csv") / 1e6
pq_mb  = os.path.getsize("events.parquet") / 1e6
print(f"CSV {csv_mb:.1f} MB  vs  Parquet {pq_mb:.1f} MB  ({csv_mb/pq_mb:.1f}x smaller)")

You can also confirm the format is doing real work with a one-liner of SQL — DuckDB will read events.parquet straight off disk, no load step:

-- Only the needed columns are read; the date filter prunes row groups.
SELECT region, COUNT(*), AVG(amount)
FROM 'events.parquet'
WHERE event_date BETWEEN DATE '2026-03-01' AND DATE '2026-03-07'
GROUP BY region
ORDER BY region;

✓ Check yourself

• Can you explain, in one sentence each, why columnar files are smaller and why they're faster for analytics?
• What do per-column min/max statistics let an engine do, and what is that called?
• Why is partitioning by user_id usually a mistake?
• Why do the techniques in this chapter matter more on object storage than on a local disk?

import pandas as pd, numpy as np, os

n = 2_000_000
df = pd.DataFrame({
    "day": pd.Timestamp("2026-06-01") + pd.to_timedelta(np.arange(n) % 30, "D"),
    "region": np.array(["us", "eu", "apac"])[np.arange(n) % 3],
    "amount": np.random.rand(n) * 100,
})
df.to_csv("d.csv", index=False)
df.to_parquet("d.parquet", compression="zstd")

size = lambda f: os.path.getsize(f) / 1e6
print(f"CSV {size('d.csv'):.1f} MB | Parquet {size('d.parquet'):.1f} MB "
      f"| {size('d.csv')/size('d.parquet'):.1f}x smaller")

Next

You can now make data small and fast on disk — but the moment a query runs, those bytes have to fit in memory and on a CPU, and that's where pipelines actually break. → Operating Systems & Resources