Start Here

What you're building

Mini-GridDP is a laptop-scale data platform for a fictional GPU-rental marketplace (the same world as the Systems Design reference). It's small enough to run on one machine, but it has every part of a real platform:

• A data generator that simulates the business — customers, GPUs, rentals, marketplace events, and a stream of GPU telemetry.
• Ingestion that lands raw data into a bronze layer, incrementally and idempotently.
• A modeled warehouse (medallion: bronze → silver → gold) with a star schema and a slowly-changing dimension.
• A telemetry rollup that tames the high-volume firehose into an hourly fact.
• Orchestration that runs it all on a schedule as a dependency-aware graph.
• A test suite, a semantic layer of metrics, a BI dashboard, CI/CD, and docs.

The architecture

Here is the whole system. Every lab adds one box or arrow to this picture:

The repo layout

Everything lives in your mini-griddp git repo (the one you started back in Course 1). By the end it looks like this — each lab fills in a folder:

mini-griddp/
├── docker-compose.yml      # the stack: Postgres, Redpanda, Dagster, Metabase  (Lab 01)
├── .env.example            # config template (real .env is gitignored)
├── README.md               # how to run it all                                 (Lab 10)
├── generator/              # synthetic data generator                          (Lab 01)
│   └── generate.py
├── ingest/                 # extract → bronze scripts                          (Lab 02, 04)
│   ├── extract_postgres.py
│   └── consume_stream.py
├── dbt/                    # dbt project: silver + gold + metrics              (Lab 03, 06, 07)
│   ├── models/
│   │   ├── staging/        # silver
│   │   └── marts/          # gold (star schema)
│   └── dbt_project.yml
├── dagster_app/            # orchestration assets                             (Lab 05)
│   └── assets.py
├── dashboards/             # Metabase notes / export                          (Lab 08)
└── .github/workflows/      # CI                                               (Lab 09)
    └── ci.yml

How the labs map to the reference

Each lab realizes, at laptop scale, a chapter of the Data Platform Systems Design guide. After a lab, read its mapped chapter to see how the same idea works at full scale:

How the labs work

These are build-along labs, not reading. Each one:

• Opens with a Setup box (what must be running) and a goal (what this lab adds to the architecture).
• Walks you through runnable code — type it, run it, and confirm each "✓ You should see" checkpoint.
• Ends with a commit step, a "✓ Check yourself", and an exercise that extends what you built.

Expect each lab to take an hour or two of focused work. Don't rush — the value is in building it yourself, hitting small problems, and fixing them. That struggle is the skill.

Prerequisites & setup

You need the full local stack from earlier courses: Docker, a Python toolchain (uv), Postgres + DuckDB, dbt, Dagster, and Redpanda — all introduced in Course 2 and Course 5. Lab 01 will have you assemble them into one Compose stack, but they should already be installed.

✓ Check yourself

• Can you describe, from the diagram, the path one rental takes from Postgres to the dashboard?
• Do you understand the repo layout and which lab fills each folder?
• Is your full stack (Docker, uv, dbt, Dagster, Redpanda) installed and working from Courses 2 and 5?

cd mini-griddp
mkdir -p generator ingest dbt dagster_app dashboards .github/workflows
touch README.md .env.example
git add -A && git commit -m "Capstone: project skeleton"

Next

Let's give the platform something to chew on: a marketplace simulator that produces realistic data. → Lab 01 — Project Setup & the Data Generator