Start Here

What "the craft" is

There's a gap between knowing how a database works and being a data engineer, and this course is that gap. The craft is the set of repeatable skills you apply every day:

• Modeling — shaping data so it's correct, queryable, and keeps its history.
• Moving — getting data in reliably, including capturing every change.
• Transforming — turning raw data into clean, business-ready tables with tested, version-controlled code.
• Storing — choosing where data lives: a warehouse, a lakehouse, a stream.
• Scheduling — running it all on time, in the right order, recovering from failure.
• Proving — testing and monitoring so people can trust the numbers.

The arc of the course

The chapters follow the path data takes through a platform, the same journey you mapped in Course 2's mental model — now you build each stage:

Modeling comes first on purpose: if the shape of your data is wrong, no amount of good tooling downstream will save you. We design, then move, then transform, then operate.

The running example

To keep the course coherent, we build around one familiar scenario: a small GPU-rental marketplace (the same world as the senior Systems Design reference, scaled down to your laptop). It has just enough richness to exercise every skill:

• A rentals source table in Postgres (customers renting GPUs by the hour) — the same one you seeded in Course 2.
• Reference data that changes over time (GPU prices, customer tiers) — perfect for slowly-changing dimensions.
• A stream of usage events — perfect for the streaming chapter.

You'll model it, ingest it, transform it through medallion layers, store it in a lakehouse, schedule it, and test it. Each chapter adds one real capability to your mini-griddp repo.

What you'll use

Everything is the free, local stack from Course 2 — now put to serious work:

If anything here is unfamiliar, it was installed in Course 2; revisit that course's setup chapters as needed.

How these chapters work

Same rhythm as before, but heavier on building. Each chapter teaches a concept, then has you do it — write the model, run the pipeline, see the result. Watch for the Setup boxes (what to have running), the runnable code blocks (type them), the "✓ You should see" checkpoints (confirm before moving on), and the "✓ Check yourself" + exercise at the end. Commit your work to mini-griddp as you go — this is the course where your portfolio really starts to take shape.

✓ Check yourself

Answer from memory before opening each one.

1. A teammate wants to skip straight to dbt and Dagster: "the tooling is so good now that we can sort out the table design later." What is this course's argument against that order?

   Show answer

   If the shape of the data is wrong, no amount of good tooling downstream will save you — every later stage inherits the model it is given. That is why the chapters run design, then move, then transform, then operate. See The arc of the course.

2. In this course's stack, which tool plays the system you ingest from, which plays the place you analyse data in, and which two tools run and version the work in between?

   Show answer

   Postgres is the source / OLTP system; DuckDB is the analytics warehouse and the lakehouse query engine. dbt holds the version-controlled SQL transformations and Dagster schedules them and manages their dependencies. See What you'll use.

3. The marketplace's finance lead asks: "what tier was this customer on at the time of each rental?" Which of the six craft skills is that a question about, and which part of the running example was chosen to exercise it?

   Show answer

   Modeling — "shaping data so it's correct, queryable, and keeps its history". The running example deliberately includes reference data that changes over time (GPU prices, customer tiers) so that the slowly-changing-dimensions chapter has something real to work on. See What "the craft" is and The running example.

4. A pipeline ran on schedule, in the right order, with no errors — and the dashboard number is still wrong. Which craft skill has done its job, and which one exists specifically so that someone other than the dashboard's reader finds this first?

   Show answer

   Scheduling did its job: on time, in order, recovered from nothing because nothing failed. A run that finishes is not a run that is right — proving (testing and monitoring, Chapter 11) is the skill that lets people trust the numbers. See What "the craft" is.

5. You run the warm-up exercise. DuckDB prints (42,), but the Postgres command fails instead of printing a count. How far into this course can you get before that matters, and what do you do?

   Show answer

   Not far: Postgres is the source for chapters 01, 05 and 06, and the rentals table seeded in Course 2 is what the pipelines ingest. Fix it before Chapter 01 by revisiting Course 2's Docker-and-databases chapter — a working stack is the prerequisite for everything here. See What you'll use and the exercise below.

# from your mini-griddp repo
docker compose up -d                       # start Postgres
docker compose exec -T postgres \
  psql -U postgres -c "SELECT count(*) FROM rentals;"   # should print a count
python -c "import duckdb; print(duckdb.sql('SELECT 42').fetchone())"  # (42,)

Next

We start where every good platform starts: with the shape of the data. First, the relational modeling and normalization that keep a transactional source clean. → Data Modeling I — Relational & Normalization