Course 4 · The Craft

The Data Engineering Craft

Now you build. Foundations explained how data systems work; this course turns that understanding into the daily craft of a data engineer — modeling data so it's trustworthy, moving it with ingestion and change-data-capture, transforming it with dbt, storing it in a lakehouse, handling streams, scheduling it all with an orchestrator, and proving it's correct with tests. Every chapter is hands-on and builds on your local stack from Course 2.

From "why" to "how"

Course 3 was concept-heavy. This one is project-heavy: you'll write real models, pipelines, and tests against the Postgres + DuckDB stack you set up in Course 2, growing your mini-griddp repo into something that looks like an actual platform. By the end you'll have all the moving parts the Capstone (Course 6) assembles into one system.

Chapters

00 Start Here What "the craft" is, the running example we build all course, and how each chapter snaps together into a real pipeline. Read first. 01 Data Modeling I — Relational & Normalization Entity-relationship modeling, primary and foreign keys, and normalization (1NF→3NF) — how to design a clean transactional schema and why normalization exists. 02 Data Modeling II — Dimensional Modeling The star schema — facts and dimensions, declaring grain, Kimball-style design — and why analytics data is modeled differently from transactional data. 03 Slowly-Changing Dimensions & History How to keep history when attributes change — SCD types 1, 2, and 3 — so you can answer "what was true at the time?" The backbone of trustworthy reporting. 04 The Medallion Architecture Bronze → silver → gold: layering raw, cleaned, and business-ready data so pipelines stay debuggable and transforms have a clear home. 05 Ingestion & ELT/ETL Getting data in — extract patterns for databases, files, and APIs; ELT vs ETL and where transforms belong; incremental extraction and idempotent loads. 06 Change Data Capture Why nightly snapshots lose history and hammer your database — and how log-based CDC streams every insert, update, and delete instead. 07 Transformation with dbt The tool that defined modern analytics engineering — models as SELECTs, refs and sources, staging→marts layering, tests, docs, and incremental models. Hands-on. 08 The Lakehouse Build a local lakehouse — Parquet files in object storage plus a table format for ACID and time travel — and query it with DuckDB. Hands-on. 09 Streaming Fundamentals Events, the log, and Kafka/Redpanda; producers and consumers; event-time vs processing-time and windows. Your first stream, hands-on. 10 Orchestration with Dagster Turn a pile of scripts into a scheduled, dependency-aware, observable pipeline — assets, schedules, dependencies, retries, and backfills. Hands-on. 11 Data Quality & Testing The trust layer — tests, freshness and volume checks, data contracts, and observability. The difference between a pipeline that runs and one people rely on.

What you'll build

By the end you'll have assembled, piece by piece, the components of a working data platform on your laptop: a modeled warehouse, ingestion with CDC, dbt transformations across medallion layers, a lakehouse, a stream, an orchestrated schedule, and a test suite. Course 6 (Capstone Labs) wires these into one end-to-end system; this course teaches each part in isolation first.