Start Here
This is the course that separates engineers who can only follow tutorials from engineers who can think. It teaches how computers, databases, and distributed systems actually work — the durable knowledge underneath every tool. This chapter explains why that matters for a data job, how the course is built, and how to get through it without burning out.
Why theory, for this job
It's tempting to skip foundations and go straight to the tools — learn dbt, learn Spark, get hired. It doesn't work, and here's the concrete reason: the hard parts of this job are never "which button do I click." They're "why did this pipeline run out of memory," "why is this query suddenly slow," "why did we get duplicate rows," "can we trust this number." Every one of those is a foundations question wearing a tool's clothing.
| A problem you'll hit | The foundation that explains it | Chapter |
|---|---|---|
| "The job crashed with OOM (out of memory)." | How memory works; why a join materializes rows | 02, 08 |
| "This query got 100× slower overnight." | Indexes, query planning, statistics | 03, 04 |
| "Reading this file is painfully slow." | Row vs columnar storage; compression | 03, 07 |
| "We're seeing duplicate events." | Delivery semantics; idempotency | 11 |
| "Two services disagree about the data." | Replication; consistency models | 10, 11 |
| "The API keeps timing out." | Networking, retries, backpressure | 09 |
Foundations are what make your knowledge transferable. Tools come and go — Hadoop gave way to Spark gave way to DuckDB and the lakehouse — but the reasons columnar storage is fast, joins are expensive, and distributed agreement is hard have not changed in decades and will outlast every tool you learn. This course is the highest-leverage time you'll spend.
The shape of the course
Twelve chapters, building from the smallest unit (a bit) up to the biggest idea (distributed agreement):
The order is deliberate: you can't understand a database (03–04) without algorithms (02); you can't understand distributed systems (10–11) without understanding a single machine (08) and the network between machines (09). Resist the urge to jump ahead.
The thread that connects it
To keep twelve chapters from feeling like twelve disconnected lectures, one question runs through all of them: "what actually happens, physically, when I work with data?" When you write SELECT … JOIN … GROUP BY, real bytes move from disk into memory, get hashed and sorted by real algorithms, across one or many real machines. By the end of this course you'll be able to narrate that journey — and that narration is the senior engineer's superpower.
We'll keep tying ideas back to the things you set up in Course 2 (Postgres, DuckDB, your first pipeline) so the theory always has somewhere concrete to land.
On "degree-like depth"
This course goes deeper than most bootcamps — into B-trees, query planners, memory hierarchies, and consensus algorithms. That's intentional: it's the material a computer-science degree would give you, filtered to what this job uses. But two promises:
- Always grounded. Every abstract idea is introduced through a concrete problem you'd actually face. No theory for theory's sake.
- No prerequisites beyond Courses 1–2. We build each idea from the ground up. Where math appears (mostly Big-O), it's explained plainly.
Some of this is genuinely deep. Aim for solid intuition on the first pass — enough to reason and to know what to look up. Mastery comes from returning to these chapters as you hit the real problems in later courses. Foundations is a reference you'll re-read, not a wall you clear once.
How to study this one
- Slow down. This isn't a course to skim. Budget the most time here (the schedule gives it ~2 months).
- Run the examples. Many chapters have small hands-on bits in Postgres/DuckDB/Python from Course 2 — running them turns abstract claims into things you've seen with your own eyes.
- Do every "✓ Check yourself." If you can't explain an idea simply, you don't have it yet — go back before moving on.
- It's fine to loop. Distributed systems (10–11) will make more sense after you've built pipelines in Course 4. Coming back is expected, not failure.
✓ Check yourself
Answer from memory before opening each one.
-
The tempting plan is: learn dbt, learn Spark, get hired, and pick up theory later. What concrete reason does this chapter give for why that plan fails on the job?
Show answer
The hard parts of the job are never "which button do I click" — they are "why did this run out of memory", "why is this query suddenly slow", "why are there duplicate rows", "can we trust this number". Each of those is a foundations question wearing a tool's clothing, so knowing the tool's buttons doesn't answer it. See Why theory, for this job.
-
A nightly job that ran fine for months starts dying with an out-of-memory error the week someone adds a new join. Which two foundations does the course point you to, and why is that more useful than searching the tool's documentation for the error?
Show answer
How memory works on a single machine (Chapter 08) and why a join materializes rows (Chapter 02). The error message names the symptom; the cause is what the join physically does with memory, and that explanation is the same in every engine — which is exactly what makes it transferable when the tool changes. See the problems table.
-
Operating systems (08) and networking (09) sit before the two distributed-systems chapters, and algorithms (02) sits before the database chapters (03–04). What is the reasoning behind that order?
Show answer
Each layer is explained in terms of the one below it. A database is built from algorithms and data structures, so 02 comes first; a distributed system is many single machines talking over a network, so you need the single machine (08) and the network between machines (09) before 10–11. The course runs small/concrete to big/abstract, which is why jumping ahead costs more than it saves. See The shape of the course.
-
One question runs through all twelve chapters. What is it, and what would a good answer to it sound like for a
SELECT … JOIN … GROUP BY?Show answer
"What actually happens, physically, when I work with data?" For that query, a good answer narrates the journey: real bytes move from disk into memory, get hashed and sorted by real algorithms, on one or many real machines. Being able to tell that story is the skill the course is building. See The thread that connects it.
-
You are two-thirds through the course, the consensus material in Chapters 10–11 is only half landing, and you are tempted either to grind on it for another month or to skip it entirely. What does the chapter advise, and what decides whether moving on is reasonable?
Show answer
Neither extreme. The target on a first pass is solid intuition — enough to reason and to know what to look up — not mastery, and the chapter says outright that distributed systems will make more sense after you've built pipelines in Course 4, so coming back is expected. Moving on is reasonable once you can explain each idea simply; if you can't, that is the signal to go back first, and skipping the chapters altogether just defers the same gap. See On "degree-like depth" and How to study this one.
Exercise — Predict before you learn (5 minutes)
Write down your current best guess at the answer to two of the "problems you'll hit" from the table above — e.g., why might a query get 100× slower overnight? Don't look anything up. Keep your guesses.
As you finish the relevant chapters, come back and compare. Whether you were right or wrong, the act of guessing first makes the real explanation stick far better — this is a well-studied learning effect (the "pretesting" effect), and it's why this curriculum keeps asking you to try before you're told.
Next
We start at the very bottom — what a piece of data physically is to a computer, and what really happens when you save a row to a file. → How Computers Represent Data