Start Here

What you're building

Mini-GridDP is a laptop-scale data platform for a fictional GPU-rental marketplace (the same world as the Systems Design reference). It's small enough to run on one machine, but it has every part of a real platform:

• A data generator that simulates the business — customers, GPUs, rentals, marketplace events, and a stream of GPU telemetry.
• Ingestion that lands raw data into a bronze layer, incrementally and idempotently.
• A modeled warehouse (medallion: bronze → silver → gold) with a star schema and a slowly-changing dimension.
• A telemetry rollup that tames the high-volume firehose into an hourly fact.
• Orchestration that runs it all on a schedule as a dependency-aware graph.
• A test suite, a semantic layer of metrics, a BI dashboard, CI/CD, and docs.

The architecture

Here is the whole system. Every lab adds one box or arrow to this picture:

The repo layout

Everything lives in your mini-griddp git repo (the one you started back in Course 1). By the end it looks like this — each lab fills in a folder:

mini-griddp/
├── docker-compose.yml      # the stack: Postgres, Redpanda, Dagster, Metabase  (Lab 01)
├── .env.example            # config template (real .env is gitignored)
├── README.md               # how to run it all                                 (Lab 10)
├── generator/              # synthetic data generator                          (Lab 01)
│   └── generate.py
├── ingest/                 # extract → bronze scripts                          (Lab 02, 04)
│   ├── extract_postgres.py
│   └── consume_stream.py
├── dbt/                    # dbt project: silver + gold + metrics              (Lab 03, 06, 07)
│   ├── models/
│   │   ├── staging/        # silver
│   │   └── marts/          # gold (star schema)
│   └── dbt_project.yml
├── dagster_app/            # orchestration assets                             (Lab 05)
│   └── assets.py
├── dashboards/             # Metabase notes / export                          (Lab 08)
└── .github/workflows/      # CI                                               (Lab 09)
    └── ci.yml

How the labs map to the reference

Each lab realizes, at laptop scale, a chapter of the Data Platform Systems Design guide. After a lab, read its mapped chapter to see how the same idea works at full scale:

How the labs work

These are build-along labs, not reading. Each one:

• Opens with a Setup box (what must be running) and a goal (what this lab adds to the architecture).
• Walks you through runnable code — type it, run it, and confirm each "✓ You should see" checkpoint.
• Ends with a commit step, a "✓ Check yourself", and an exercise that extends what you built.

Expect each lab to take an hour or two of focused work. Don't rush — the value is in building it yourself, hitting small problems, and fixing them. That struggle is the skill.

Prerequisites & setup

You need the full local stack from earlier courses: Docker, a Python toolchain (uv), Postgres + DuckDB, dbt, Dagster, and Redpanda — all introduced in Course 2 and Course 5. Lab 01 will have you assemble them into one Compose stack, but they should already be installed.

✓ Check yourself

Answer from memory before opening each one.

1. A customer rents an H100 this morning. Name, in order, every hop that rental takes before it shows up as part of a number on the dashboard.

   Show answer

   The row is written to Postgres rentals; the Python ingest scripts land it raw in bronze; dbt cleans it into silver and models it into fct_rentals in the gold star schema; the semantic layer turns that fact into a metric; Metabase charts the metric. Bronze, silver and gold all live in the one DuckDB warehouse. See The architecture.

2. GPU telemetry does not follow the same route as rentals. Where does it enter, what happens to it before it reaches gold, and why is it treated differently?

   Show answer

   It arrives through Redpanda and is read by the stream consumer, not the Postgres extract, and what reaches gold is a rollup — fct_telemetry_hourly — rather than the raw readings. Telemetry is the platform's high-volume firehose, so it is summarised into an hourly fact small enough to model and serve. See What you're building and the diagram.

3. In the diagram, Dagster, the dbt tests and CI sit in a band underneath the flow instead of being boxes the data passes through. What does that placement tell you about their job?

   Show answer

   None of them stores or moves data. Dagster runs the boxes above — ingest, then dbt, then rollups — in dependency order on a schedule; tests and freshness checks judge what those steps produced; CI checks changes to the code. They cut across every stage rather than being one. See The architecture.

4. You clone mini-griddp onto a new laptop and the stack complains about missing configuration, although it runs fine on your old machine. Going by the repo layout, what is missing and why was it never in the clone?

   Show answer

   The real .env. Only .env.example — the template — is committed; the real file is gitignored so credentials never enter the history. Copy the template to .env and fill it in. See The repo layout.

5. A friend is fluent in SQL and Docker and has written a few dbt models, but has never defined a Dagster asset. They ask whether to start Lab 01 today or go back to Course 5 first. What do you tell them, and what would change your answer?

   Show answer

   The labs move fast and don't re-teach basics, so every gap has to be closed somewhere — what decides it is how many there are and how early they bite. Dagster isn't needed until Lab 05, so that single gap can be filled while Labs 01–04 are under way. If Compose or dbt were the shaky ones the advice flips: they are needed from Lab 01 and Lab 03, so the earlier courses come first. See Prerequisites and which lab fills which folder.

cd mini-griddp
mkdir -p generator ingest dbt dagster_app dashboards .github/workflows
touch README.md .env.example
git add -A && git commit -m "Capstone: project skeleton"

Next

Let's give the platform something to chew on: a marketplace simulator that produces realistic data. → Lab 01 — Project Setup & the Data Generator