Start Here

What "the craft" is

There's a gap between knowing how a database works and being a data engineer, and this course is that gap. The craft is the set of repeatable skills you apply every day:

• Modeling — shaping data so it's correct, queryable, and keeps its history.
• Moving — getting data in reliably, including capturing every change.
• Transforming — turning raw data into clean, business-ready tables with tested, version-controlled code.
• Storing — choosing where data lives: a warehouse, a lakehouse, a stream.
• Scheduling — running it all on time, in the right order, recovering from failure.
• Proving — testing and monitoring so people can trust the numbers.

The arc of the course

The chapters follow the path data takes through a platform, the same journey you mapped in Course 2's mental model — now you build each stage:

Modeling comes first on purpose: if the shape of your data is wrong, no amount of good tooling downstream will save you. We design, then move, then transform, then operate.

The running example

To keep the course coherent, we build around one familiar scenario: a small GPU-rental marketplace (the same world as the senior Systems Design reference, scaled down to your laptop). It has just enough richness to exercise every skill:

• A rentals source table in Postgres (customers renting GPUs by the hour) — the same one you seeded in Course 2.
• Reference data that changes over time (GPU prices, customer tiers) — perfect for slowly-changing dimensions.
• A stream of usage events — perfect for the streaming chapter.

You'll model it, ingest it, transform it through medallion layers, store it in a lakehouse, schedule it, and test it. Each chapter adds one real capability to your mini-griddp repo.

What you'll use

Everything is the free, local stack from Course 2 — now put to serious work:

If anything here is unfamiliar, it was installed in Course 2; revisit that course's setup chapters as needed.

How these chapters work

Same rhythm as before, but heavier on building. Each chapter teaches a concept, then has you do it — write the model, run the pipeline, see the result. Watch for the Setup boxes (what to have running), the runnable code blocks (type them), the "✓ You should see" checkpoints (confirm before moving on), and the "✓ Check yourself" + exercise at the end. Commit your work to mini-griddp as you go — this is the course where your portfolio really starts to take shape.

✓ Check yourself

• Can you name the six skills that make up "the craft"?
• Do you see why modeling comes before moving and transforming?
• Is your Course 2 stack (Postgres + DuckDB, your mini-griddp repo) still working? If not, fix that first.

# from your mini-griddp repo
docker compose up -d                       # start Postgres
docker compose exec -T postgres \
  psql -U postgres -c "SELECT count(*) FROM rentals;"   # should print a count
python -c "import duckdb; print(duckdb.sql('SELECT 42').fetchone())"  # (42,)

Next

We start where every good platform starts: with the shape of the data. First, the relational modeling and normalization that keep a transactional source clean. → Data Modeling I — Relational & Normalization