Files & Serialization at Scale

The format is a decision, not an afterthought

Beginners reach for CSV by reflex — it opens in a spreadsheet, it's human-readable, every tool accepts it. For small files that's fine. But the moment your data grows, the file format becomes a performance and cost decision, and CSV is usually the wrong one.

Here's the concrete stake. In Chapter 01 you saw that a file is just serialized bytes. In Chapter 03 you met the distinction between row-oriented and columnar storage inside a database. File formats make exactly the same choice — but now you can see and touch the result, because it's a file on disk you can measure. A dashboard that scans a 50 GB CSV every time someone clicks a filter will be slow and run up a cloud bill; the identical data in partitioned, compressed Parquet might scan 2 GB and cost a tenth as much. Nothing about the data changed — only its physical layout.

Row formats vs columnar formats

Every file format has to decide one thing: as it lays bytes down on disk, does it keep a whole record together, or a whole column together? That single choice splits the format world in two.

• Row-oriented: CSV, JSON (lines), Avro. Values for one record sit next to each other.
• Columnar: Parquet, ORC. Values for one column sit next to each other.

Picture a tiny table of three rows and three columns, and look at the byte order each layout writes:

Why does columnar make files smaller? Because similar values end up adjacent. A column of region values is just us, eu, us, us, eu… — lots of repetition, which compresses brilliantly. A column of timestamps increases steadily, so you can store the differences instead of full values. When you mix all three columns together (the row layout), the byte stream looks random and compresses poorly.

Why does columnar make analytics faster? Analytics queries are wide and shallow: SELECT region, SUM(amount) … GROUP BY region touches 2 of maybe 200 columns, but every row. In a columnar file the engine reads only the region and amount chunks and skips the other 198 entirely. In a row file it must read every byte of every row to get at those two fields.

Inside a Parquet file

Parquet is the default analytics format you'll meet everywhere — DuckDB, Spark, BigQuery, Snowflake all read and write it. It's worth understanding its insides, because they explain why a well-targeted query barely touches the disk.

A Parquet file is a nested structure, not a flat dump:

Three things in there change the game:

• Embedded schema and types. Unlike CSV — where everything is text and "2026-06-21" might be a date or a string, you can't tell — Parquet stores the real type of each column. An engine reads a Parquet file already knowing column 3 is a 64-bit integer. No guessing, no parsing dates out of strings.
• Column chunks. Because each column lives in its own chunk, an engine seeks straight to the chunks it needs and reads nothing else. This is the file-level version of the columnar advantage above.
• Per-column statistics. The footer records the min and max value of each column within each row group. This is the quiet superpower.

Those statistics enable predicate pushdown (also called row-group skipping). Suppose a row group's date column has min = 2026-01-01 and max = 2026-01-31, and your query says WHERE date = '2026-06-21'. The engine reads just the footer, sees that June 21 cannot possibly be in a January-only row group, and skips the entire row group without reading it. On a file with hundreds of row groups, a selective filter might touch only one or two — that's how "WHERE date = X" stays cheap even on huge files.

Compression: the CPU-vs-size tradeoff

On top of the columnar layout, Parquet compresses each page with a codec. Compression shrinks the bytes on disk (less storage cost, fewer bytes to read from disk or network) at the price of CPU time to compress when writing and decompress when reading. That's the whole tradeoff: smaller files cost more CPU. The art is picking where to sit on that curve.

Columnar data compresses far better than row data for the reason we saw: a column is full of similar, often repeated values, and compression algorithms feed on repetition. The same gzip applied to a CSV might cut it in half; applied to columnar pages it can cut it by 5×.

Partitioning and the small-files problem

Row-group skipping happens inside a file. Partitioning applies the same idea one level up — across files. You split a dataset into a folder tree keyed by a column, most often a date:

Now a query filtered on date never even opens the other days' files — the engine prunes them by reading the directory names. This is wildly effective for the most common analytics filter (a date range), which is why date partitioning is the default move.

But there's a trap, and it's the one beginners fall into: the small-files problem. If you partition too finely — say by minute, or by date and region and user — you end up with millions of tiny files. Every file carries fixed overhead (open it, read its footer, list it in the directory), so thousands of 10 KB files are dramatically slower to scan than a handful of 256 MB files holding the same rows. Object storage especially punishes many small reads.

The fix when small files accumulate is compaction: a periodic job that reads many small files in a partition and rewrites them as a few large ones.

Object storage: the substrate of the lake

Where do all these Parquet files live? Increasingly, in object storage — Amazon S3, Google Cloud Storage, or the MinIO server you stood up locally in Course 2. Object storage is the substrate that makes the modern data platform possible:

• Cheap. Far cheaper per gigabyte than database storage or attached disks.
• Durable. Providers replicate your bytes across machines and facilities; data loss is vanishingly rare.
• Effectively infinite. You never run out of space or manage volumes — you just write more objects.

The catch: object storage has higher latency than a local disk. Each object is fetched over the network via an HTTP request, so a single read is slow to start, even though throughput once flowing is high. This is precisely why the techniques above matter so much here — column chunks, row-group skipping, and partition pruning all minimize the number and size of reads, which is exactly what high-latency object storage rewards.

Hands-on: measure CSV vs Parquet yourself

Claims about "5–10× smaller and faster" only stick once you've seen them. Let's generate a dataset, write it both ways, and compare. We'll use DuckDB from Python (installed in Course 2; if not, pip install duckdb). DuckDB is itself a columnar engine and reads/writes Parquet natively.

import duckdb, os, time

con = duckdb.connect()

# 1) Build a realistic-ish table: 5 million rows, a few repetitive columns.
con.execute("""
    CREATE TABLE events AS
    SELECT
        i                                   AS id,
        DATE '2026-01-01' + (i % 180)::INTEGER AS event_date, -- 180 distinct days (DATE + n needs INTEGER)
        ['us','eu','apac'][1 + (i % 3)]     AS region,        -- only 3 values
        (i * 7 % 1000) / 10.0               AS amount
    FROM range(5_000_000) t(i);
""")

# 2) Write the SAME data two ways.
con.execute("COPY events TO 'events.csv'     (FORMAT CSV, HEADER);")
con.execute("COPY events TO 'events.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);")

# 3) Compare file sizes.
csv_mb = os.path.getsize('events.csv') / 1e6
pq_mb  = os.path.getsize('events.parquet') / 1e6
print(f"CSV    : {csv_mb:8.1f} MB")
print(f"Parquet: {pq_mb:8.1f} MB   ({csv_mb / pq_mb:.1f}x smaller)")

# 4) Compare scan speed for a typical analytics query.
def timed(sql):
    start = time.perf_counter()
    con.execute(sql).fetchall()
    return time.perf_counter() - start

q = "SELECT region, SUM(amount) FROM read_{}('events.{}') WHERE event_date = DATE '2026-03-01' GROUP BY region"
t_csv = timed(q.format('csv', 'csv'))
t_pq  = timed(q.format('parquet', 'parquet'))
print(f"\nScan CSV    : {t_csv*1000:7.1f} ms")
print(f"Scan Parquet: {t_pq*1000:7.1f} ms   ({t_csv / t_pq:.1f}x faster)")

Prefer pandas? The same comparison, fewer moving parts:

import pandas as pd, numpy as np, os

n = 5_000_000
df = pd.DataFrame({
    "id": np.arange(n),
    "event_date": pd.Timestamp("2026-01-01") + pd.to_timedelta(np.arange(n) % 180, "D"),
    "region": np.array(["us", "eu", "apac"])[np.arange(n) % 3],
    "amount": (np.arange(n) * 7 % 1000) / 10.0,
})

df.to_csv("events.csv", index=False)
df.to_parquet("events.parquet", compression="zstd")   # needs pyarrow

csv_mb = os.path.getsize("events.csv") / 1e6
pq_mb  = os.path.getsize("events.parquet") / 1e6
print(f"CSV {csv_mb:.1f} MB  vs  Parquet {pq_mb:.1f} MB  ({csv_mb/pq_mb:.1f}x smaller)")

You can also confirm the format is doing real work with a one-liner of SQL — DuckDB will read events.parquet straight off disk, no load step:

-- Only the needed columns are read; the date filter prunes row groups.
SELECT region, COUNT(*), AVG(amount)
FROM 'events.parquet'
WHERE event_date BETWEEN DATE '2026-03-01' AND DATE '2026-03-07'
GROUP BY region
ORDER BY region;

✓ Check yourself

Answer from memory before opening each one.

1. A query computes SUM(amount) … GROUP BY region over every row of a 200-column dataset. What does an engine have to read from a CSV copy versus a Parquet copy, and why is the Parquet copy also several times smaller on disk before the query even starts?

   Show answer

   From the CSV it must read and parse every byte of every row to reach two fields; from Parquet it seeks to the region and amount column chunks and skips the other 198. The file is smaller because a column's values are similar and repetitive (us, eu, us…, steadily rising timestamps), which compression feeds on, whereas interleaved row data looks random and compresses poorly. See Row formats vs columnar formats.

2. A Parquet file holds 180 days of events written in arrival order from a shuffled backfill, so every row group contains dates from across the whole range. You run WHERE event_date = '2026-03-01'. How many row groups can the engine skip, and what would have to be true of the file for the filter to become cheap?

   Show answer

   None. Skipping works by reading each row group's min and max from the footer and ruling out groups that cannot contain the value; here every group's range runs from the first day to the last, so none can be ruled out. The tempting belief is that Parquet makes any filter cheap — the statistics only help when the data is sorted or clustered on the filter column (or partitioned by it), so that each row group covers a narrow range. You still get the columnar benefit of reading only the columns you need. See Inside a Parquet file.

3. To "make every filter fast", a team partitions its events by date, hour and region. The dataset is now hundreds of thousands of files of about 10 KB each, and dashboards are slower than before. Why, and what are the two fixes?

   Show answer

   This is the small-files problem: each file carries fixed overhead — list it, open it, read its footer — so thousands of tiny files scan far more slowly than a handful of large ones holding the same rows, and object storage punishes many small reads hardest. Partition coarsely, on a low-cardinality column you actually filter on (date), aiming for files of tens to hundreds of megabytes, and run compaction to rewrite the accumulated small files into a few big ones. See Partitioning and the small-files problem.

4. The same Parquet dataset is queried from a local SSD and from S3. Why do column chunks, row-group skipping and partition pruning matter even more in the second case?

   Show answer

   Every object-storage read is an HTTP request over the network: slow to start, though throughput is high once bytes flow. The cost of a query is therefore driven by how many reads it makes and how large they are, and all three techniques exist to cut exactly that — they let the engine decide what not to fetch. On a low-latency local disk a wasted read is cheap; on S3 each one is a round trip. See Object storage.

5. A service emits one event at a time onto a stream, around the clock; analysts later aggregate months of those events. Someone proposes Parquet everywhere "because it's the analytics format". Which format belongs at each end, and what decides it?

   Show answer

   A row format such as Avro on the stream, Parquet for the analytics tables. What decides it is the access pattern: appending whole records one at a time is cheap in a row format, while a columnar file would have to touch every column to add a single row. Scanning a few columns across many rows is where columnar wins on both size and speed. So row for whole-record writes and streaming, columnar for analytics over many rows — and converting from one to the other is a normal pipeline step. See When row formats win.

6. You are choosing a Parquet compression codec for two datasets: a hot table scanned all day by a busy cluster, and a cold archive that is rarely read and billed by the gigabyte. What are you trading when you pick a codec, and what would you choose for each?

   Show answer

   The trade is CPU for bytes: smaller files cost more CPU to write and to decompress on every read. For the hot table, favour decompression speed — Snappy (the default) or Zstd, which gets near-gzip ratios at near-Snappy speed. For the archive, size matters most and read speed least, so a high-ratio codec fits: Zstd at a higher level, or gzip if something downstream demands maximum portability. When unsure, measure on your own data rather than agonise. See Compression.

import pandas as pd, numpy as np, os

n = 2_000_000
df = pd.DataFrame({
    "day": pd.Timestamp("2026-06-01") + pd.to_timedelta(np.arange(n) % 30, "D"),
    "region": np.array(["us", "eu", "apac"])[np.arange(n) % 3],
    "amount": np.random.rand(n) * 100,
})
df.to_csv("d.csv", index=False)
df.to_parquet("d.parquet", compression="zstd")

size = lambda f: os.path.getsize(f) / 1e6
print(f"CSV {size('d.csv'):.1f} MB | Parquet {size('d.parquet'):.1f} MB "
      f"| {size('d.csv')/size('d.parquet'):.1f}x smaller")

Takeaway

• CSV is a good interchange format and a poor analytics format, because an engine must read every byte and parse text to learn types.
• Columnar files keep each column's values together, so a query that touches 2 of 200 columns reads only those chunks and compresses well on repeated values.
• A Parquet footer holds the schema and the min and max of each column per row group, so the engine can skip row groups that cannot match the filter.
• Partitioning by a low-cardinality column you filter on, usually date, lets the engine prune whole files. Partitioning too finely creates small files whose open and read overhead outweighs the data.
• Fix small files with compaction, and aim for files in the tens to hundreds of megabytes.
• Snappy is the safe default codec and Zstd gets smaller files at a modest read cost. Row formats such as Avro still win for appending whole records.
• Files in object storage read by an engine are a data lake: cheap and durable, with higher latency per request, which is why layout choices matter.

Files are only useful once they fit in memory and on a CPU, and that is where pipelines break. → Operating Systems & Resources

Further reading

• Parquet file format — Apache Parquet documentation. Row groups, column chunks, pages and the footer metadata.
• Parquet files — DuckDB documentation. Reading and writing Parquet, including the compression options used in the lab.
• Parquet tips — DuckDB documentation. Row group size, filter pushdown and how to write files that query well.
• Hive partitioning — DuckDB documentation. How folder names become columns and let the engine prune files.
• Dremel: Interactive Analysis of Web-Scale Datasets — Melnik et al., Google, VLDB 2010. The columnar nested-data design that Parquet descends from.