Lab 10 — Ship & Document

The goal

Up to now, the audience for your repo has been you. You know which command to run, which port the dashboard is on, why the telemetry rollup exists. That knowledge lives in your head. The goal of this lab is to move it into the repo so that a hiring manager — or future-you in six months — can clone it, run it, and understand it without your help.

A portfolio data project is judged on three things, and only three:

You already did the hard engineering. This lab is about the last 10% that makes the other 90% legible to someone else.

Write a great README

The README is the front door. Most people will read it and never run anything — so it has to show the platform, not just describe it. A strong README has five parts: a one-line description, an architecture diagram, the stack, "what it does," and run-from-scratch instructions.

Here's the architecture again — the same diagram from Start Here, now as the centerpiece of your README. Drop it into a fenced code block so it renders as monospace on GitHub:

One small file first. Every script in the repo has a working default for the Compose stack, and .env is gitignored, so document the knobs in a committed .env.example rather than asking a stranger to create config they do not need:

# Optional overrides. Every script defaults to the Compose stack below;
# export these (for example `set -a; source .env`) only if your stack differs.
GRIDDP_DSN=postgresql://griddp:griddp@localhost:5432/marketplace     # generator/generate.py
PG_CONN="host=localhost port=5432 dbname=marketplace user=griddp password=griddp"   # ingest/extract_postgres.py
WAREHOUSE=warehouse.duckdb                                            # ingest/extract_postgres.py
SERVING_PG_DSN=postgresql://griddp:griddp@localhost:5432/marketplace  # ingest/publish_to_serving.py
DUCKDB_PATH=warehouse.duckdb                                          # ingest/publish_to_serving.py

Now write the README. Open README.md in your repo root and use this as a template — fill in the bracketed bits and keep the structure:

# mini-GridDP

A laptop-scale data platform for a fictional GPU-rental marketplace — ingestion,
a medallion warehouse, orchestration, tests, a semantic layer, and a BI dashboard,
all runnable with one Compose stack.

## Architecture

```text
  SOURCES            INGEST        STORE / MODEL (DuckDB)        SERVE
  Postgres  ──┐   ┌─────────┐    ┌──────────────────────┐    ┌──────────┐
  rentals,    ├──▶│ ingest  │───▶│ BRONZE raw landed     │──▶ │ semantic │
  customers,  │   │ scripts │    │ SILVER cleaned + SCD2 │    │ layer    │
  gpus,prices │   └─────────┘    │ GOLD   star schema    │    └────┬─────┘
  Redpanda  ──┘   ┌─────────┐    └──────────────────────┘         ▼
  events +        │ stream  │──────── rollup ──────▲          ┌──────────┐
  TELEMETRY ⚡     │ consumer│   dbt transforms ────┘          │ Metabase │
                  └─────────┘                                 └──────────┘

  Orchestration: Dagster  ·  Quality: dbt tests + freshness  ·  CI: GitHub Actions
```

## Stack

| Layer          | Tool                          |
|----------------|-------------------------------|
| Sources        | Postgres, Redpanda (Kafka API)|
| Ingestion      | Python (extract + stream)     |
| Warehouse      | DuckDB                        |
| Transformation | dbt (bronze → silver → gold)  |
| Orchestration  | Dagster                       |
| Quality        | dbt tests + freshness checks  |
| Serving / BI   | semantic metrics + Metabase   |
| CI/CD          | GitHub Actions                |

## What it does

- Simulates a GPU-rental marketplace: customers, GPUs, rentals, a price
  history, and a high-volume stream of GPU telemetry.
- Lands raw data into a **bronze** layer incrementally and idempotently.
- Models a **medallion** warehouse: silver (cleaned, with an SCD2 price
  dimension) and gold (a star schema: `dim_customer`, `dim_gpu`, `dim_date`,
  `fct_rentals`, `fct_telemetry_hourly`).
- Rolls the telemetry firehose up into an hourly fact.
- Orchestrates ingest → dbt → rollups as a scheduled Dagster graph.
- Tests the data and exposes governed metrics to a Metabase dashboard.

## Run it from scratch

Requires Docker and [uv](https://docs.astral.sh/uv/) (plus the `duckdb` CLI,
same minor version as the `duckdb` package in `pyproject.toml`, if you want to
poke at the warehouse). One sequence, top to bottom:

```bash
git clone https://github.com/<you>/mini-griddp.git
cd mini-griddp
docker compose up -d                              # Postgres, Redpanda, Dagster placeholder, Metabase
docker compose ps                                 # wait for postgres + redpanda to show (healthy)
uv run python generator/generate.py               # seed the Postgres source (uv builds the venv first time)
uv run python generator/telemetry_producer.py     # let it run a minute, then Ctrl-C
uv run dagster job execute -j full_pipeline -m dagster_app.definitions   # ingest → dbt → rollup
uv run python ingest/publish_to_serving.py        # gold + metrics → Postgres `serving` schema
```

Then open Metabase at **http://localhost:3001**, connect it to Postgres
(host `postgres`, database `marketplace`, schema `serving`) and rebuild the
dashboard from the SQL in `dashboards/`. For the Dagster UI, free port 3000
from the placeholder and run the real thing:
`docker compose stop dagster && uv run dagster dev`, then **http://localhost:3000**.

## Project layout

See `mini-griddp/` — `generator/`, `ingest/`, `dbt/`, `dagster_app/`,
`dashboards/`, `.github/workflows/`.

## Notes

Built as the capstone of a data-platform curriculum. Each component maps to a
chapter of a full-scale systems-design reference; this is the laptop-scale
realization of that design.

Adjust the command names if your repo drifted from the labs (a renamed Dagster job, an extra generator flag). The template is a skeleton — the value is that every command in it really works, which we verify next.

The run-from-scratch runbook

The README's run section is a promise. Now keep it. The only honest way to know your instructions work is to run them as a stranger would — from a clean clone, in a fresh directory, following only what's written. Do exactly that:

# 1. Clone into a throwaway directory — pretend you've never seen this repo
cd /tmp
rm -rf griddp-check
git clone https://github.com/<you>/mini-griddp.git griddp-check
cd griddp-check

# 2. Bring up the stack
docker compose up -d
docker compose ps              # postgres and redpanda show (healthy); the rest "running"

# 3. Seed the sources: Postgres rows, then a minute of telemetry
uv run python generator/generate.py
uv run python generator/telemetry_producer.py    # Ctrl-C after a minute

# 4. Build the platform: ingest -> dbt -> rollups, then publish for BI
uv run dagster job execute -j full_pipeline -m dagster_app.definitions
uv run python ingest/publish_to_serving.py

# 5. Open the results
open http://localhost:3001     # Metabase  (xdg-open on Linux)
docker compose stop dagster && uv run dagster dev   # then open http://localhost:3000

Presenting it as a portfolio piece

A project nobody sees can't get you hired. Three moves turn the repo into something an employer can find, trust, and remember.

1. Push it with a clean history. You've been committing per lab — that history is the story of building a platform, and it's a feature. Make sure the public repo has a good description, the README as its landing page, and no secrets in any commit (check that .env was always gitignored; if a secret ever slipped in, rotate it and scrub history). Pin the repo on your GitHub profile.

2. Write a short writeup. A blog post or a LinkedIn post, 300–600 words: what you built, a screenshot of the dashboard and the architecture diagram, and the two or three decisions you're proudest of (why a medallion layout, why you rolled telemetry up hourly, why Dagster over cron). Link the repo. This is what turns a recruiter scroll into a click.

3. Practice a two-minute verbal walkthrough. You will be asked "tell me about a project" in every interview. Rehearse the answer out loud until it's smooth:

• 0:00–0:20 — what & why. "Mini-GridDP is an end-to-end data platform for a GPU-rental marketplace. I built it to practice every layer of a real platform on one machine."
• 0:20–1:20 — the path of one record. Trace a rental from Postgres → bronze → silver (priced through the SCD2 price dimension) → the gold star schema → a metric → a dashboard chart. The data-flow story shows you understand the whole, not just pieces.
• 1:20–2:00 — a decision and a tradeoff. Pick one thing you chose and one thing you'd do differently at scale (e.g. "DuckDB is perfect on a laptop; in production I'd swap the gold layer onto a warehouse and keep the same dbt models").

A roadmap of extensions

A portfolio piece is more compelling when it's clearly alive — when the README ends with "next steps" and you've actually shipped one or two. Each of these reuses the structure you already have, so the marginal effort is small and the learning is real. Add a short "Roadmap" section to your README listing the ones you plan to do.

Reflect: you built a platform

Step back. Across ten labs you assembled, by hand, every layer of a production data platform — and each piece maps directly to a chapter of the Data Platform Systems Design reference:

Wrapping up the capstone

That's Course 6. You didn't read about a data platform — you built one, end to end, and made it reproducible enough that a stranger can run it. This repo is your proof: concrete, runnable, and yours.

The work isn't to keep building forever. It's to use what you built. A great project sitting in a private repo helps no one — least of all you. The next course is about taking this exact artifact into the job market: Course 7 — Career & Getting the Job turns your platform, your writeup, and your two-minute walkthrough into résumé bullets, interview stories, and offers.

Final commit & tag

Commit the README and any fixes the runbook surfaced, then mark this milestone with a version tag. A tag turns "a repo" into "a release" — a clear, citable point that says this is done and it works.

git add README.md .env.example
git commit -m "Lab 10: README, runbook, and roadmap — platform documented"

# tag the release
git tag -a v1.0 -m "mini-GridDP v1.0 — complete, runnable, documented"

git push origin main
git push origin v1.0

✓ Check yourself

Answer from memory before opening each one.

1. Your runbook passes every time you run it inside your working copy. Name three pieces of state that exist there and will not exist in a stranger's fresh clone.

   Show answer

   Anything gitignored or outside git altogether: the real .env, the built warehouse.duckdb, the already-seeded Postgres volume, Metabase's saved setup, your virtualenv and installed tool versions. A working copy cannot reveal a missing step because that state fills the gap — only a clean clone in a fresh directory tests the README. See The run-from-scratch runbook.

2. A friend following the README gets "connection refused" from the seed step immediately after docker compose up -d; thirty seconds later the same command works. Whose bug is it, and what is the fix?

   Show answer

   The README's. Postgres needs a few seconds to become healthy and the instructions let the reader race it. Add the wait — confirm docker compose ps shows healthy before seeding, or make the step wait on the health check. Any failure in the documented sequence is a documentation bug, not a nuisance. See The classic gaps.

3. After a clean clone and a green Dagster run, a stranger opens Metabase. Which part of the platform is the repo unable to recreate by itself, and what did you commit so that it isn't lost?

   Show answer

   The dashboard. Metabase keeps its connections, questions and dashboards in its own database, not in git, so a fresh clone gets an unconfigured Metabase however clean the pipeline run was. Lab 08 commits each question's SQL plus notes on chart type and layout under dashboards/, which makes it rebuildable in minutes — and the README should say so as a step. See the runbook.

4. Checking the history, you find your real .env was committed in Lab 02 and deleted in Lab 03. The current tree is clean. What still has to happen before the repo goes public, and in which order?

   Show answer

   Deleting a file only removes it from the latest commit; it is still readable in history. Rotate the secret first — anything that has been in a pushed commit should be treated as compromised — then scrub it from history and confirm .env is gitignored. See Push it with a clean history.

5. During the two-minute walkthrough an interviewer interrupts: "DuckDB? That wouldn't survive production." What shape of answer does the chapter prepare you to give?

   Show answer

   Agree, and show you chose it knowingly — a decision plus a tradeoff. DuckDB is the right engine for one laptop; at scale you would move the gold layer onto a warehouse and keep the same dbt models. Defending the tool is weaker than showing which parts of the design survive the swap. See the two-minute walkthrough.

6. You have two free weeks and six candidate extensions. How do you choose, and when is doing more than one justified?

   Show answer

   One extension built end to end and written up beats a list of half-finished branches. Choose by setting the "stretches you on" column against the roles you are applying for — the cloud deploy for a cloud-warehouse shop, streaming SCD for a CDC-heavy one. A second is justified only once the first is shipped and documented. See A roadmap of extensions.

Takeaway

• A README's run section is a promise, and the only honest test is to follow it from a clean clone in a fresh directory as a stranger would.
• Hidden state is what breaks a clean-clone run: a committed secret instead of .env.example, an assumed tool version, or a service that needs seconds to become healthy.
• Put the architecture diagram high in the README, because a reviewer decides in ten seconds whether to keep reading.
• A platform that runs from a clean clone and that you can explain decision by decision beats a flashier project nobody can reproduce or you cannot defend.
• Pick one extension and build it end to end, since a focused follow-up shows initiative and a list of six unfinished ones does not.
• A two-minute walkthrough covers what and why, the path of one record from Postgres to a chart, and one decision with the trade-off at scale.
• A version tag turns the repository into a release, a clear point that says this is done and it works.

The build is finished, and the next course turns this repository into a résumé, an interview story and an offer. → Career & Getting the Job

Further reading

• GitHub: About the repository README file — GitHub docs, accessed 2026. What a README should hold and where GitHub shows it.
• GitHub: Managing releases in a repository — GitHub docs, accessed 2026. Turning a tag into a release.
• Pro Git: Tagging — Chacon and Straub, Apress, free online. Annotated and lightweight tags and how to push them.
• Semantic Versioning 2.0.0 — Preston-Werner. What a version number such as v1.0 promises.