Lab 10 — Ship & Document

The goal

Up to now, the audience for your repo has been you. You know which command to run, which port the dashboard is on, why the telemetry rollup exists. That knowledge lives in your head. The goal of this lab is to move it into the repo so that a hiring manager — or future-you in six months — can clone it, run it, and understand it without your help.

A portfolio data project is judged on three things, and only three:

You already did the hard engineering. This lab is about the last 10% that makes the other 90% legible to someone else.

Write a great README

The README is the front door. Most people will read it and never run anything — so it has to show the platform, not just describe it. A strong README has five parts: a one-line description, an architecture diagram, the stack, "what it does," and run-from-scratch instructions.

Here's the architecture again — the same diagram from Start Here, now as the centerpiece of your README. Drop it into a fenced code block so it renders as monospace on GitHub:

Now write the README. Open README.md in your repo root and use this as a template — fill in the bracketed bits and keep the structure:

# mini-GridDP

A laptop-scale data platform for a fictional GPU-rental marketplace — ingestion,
a medallion warehouse, orchestration, tests, a semantic layer, and a BI dashboard,
all runnable with one Compose stack.

## Architecture

```text
  SOURCES            INGEST        STORE / MODEL (DuckDB)        SERVE
  Postgres  ──┐   ┌─────────┐    ┌──────────────────────┐    ┌──────────┐
  rentals,    ├──▶│ ingest  │───▶│ BRONZE raw landed     │──▶ │ semantic │
  customers,  │   │ scripts │    │ SILVER cleaned + SCD2 │    │ layer    │
  gpus,prices │   └─────────┘    │ GOLD   star schema    │    └────┬─────┘
  Redpanda  ──┘   ┌─────────┐    └──────────────────────┘         ▼
  events +        │ stream  │──────── rollup ──────▲          ┌──────────┐
  TELEMETRY ⚡     │ consumer│   dbt transforms ────┘          │ Metabase │
                  └─────────┘                                 └──────────┘

  Orchestration: Dagster  ·  Quality: dbt tests + freshness  ·  CI: GitHub Actions
```

## Stack

| Layer          | Tool                          |
|----------------|-------------------------------|
| Sources        | Postgres, Redpanda (Kafka API)|
| Ingestion      | Python (extract + stream)     |
| Warehouse      | DuckDB                        |
| Transformation | dbt (bronze → silver → gold)  |
| Orchestration  | Dagster                       |
| Quality        | dbt tests + freshness checks  |
| Serving / BI   | semantic metrics + Metabase   |
| CI/CD          | GitHub Actions                |

## What it does

- Simulates a GPU-rental marketplace: customers, GPUs, rentals, prices,
  marketplace events, and a high-volume stream of GPU telemetry.
- Lands raw data into a **bronze** layer incrementally and idempotently.
- Models a **medallion** warehouse: silver (cleaned, with an SCD2 customer
  dimension) and gold (a star schema: `dim_customer`, `dim_gpu`, `dim_date`,
  `fct_rentals`, `fct_telemetry_hourly`).
- Rolls the telemetry firehose up into an hourly fact.
- Orchestrates ingest → dbt → rollups as a scheduled Dagster graph.
- Tests the data and exposes governed metrics to a Metabase dashboard.

## Run it from scratch

Requires Docker and [uv](https://docs.astral.sh/uv/). One sequence, top to bottom:

```bash
git clone https://github.com/<you>/mini-griddp.git
cd mini-griddp
cp .env.example .env            # defaults work as-is for local
docker compose up -d            # Postgres, Redpanda, Dagster, Metabase
uv run python generator/generate.py --seed   # seed sources
uv run dagster job execute -m dagster_app -j build_platform  # ingest → dbt → rollup
```

Then open the dashboard at **http://localhost:3000** (Metabase) and the
orchestration UI at **http://localhost:3001** (Dagster).

## Project layout

See `mini-griddp/` — `generator/`, `ingest/`, `dbt/`, `dagster_app/`,
`dashboards/`, `.github/workflows/`.

## Notes

Built as the capstone of a data-platform curriculum. Each component maps to a
chapter of a full-scale systems-design reference; this is the laptop-scale
realization of that design.

Adjust the command names to match what your repo actually uses (your Dagster job name, your generator flags). The template is a skeleton — the value is that every command in it really works, which we verify next.

The run-from-scratch runbook

The README's run section is a promise. Now keep it. The only honest way to know your instructions work is to run them as a stranger would — from a clean clone, in a fresh directory, following only what's written. Do exactly that:

# 1. Clone into a throwaway directory — pretend you've never seen this repo
cd /tmp
rm -rf griddp-check
git clone https://github.com/<you>/mini-griddp.git griddp-check
cd griddp-check

# 2. Config from the template
cp .env.example .env

# 3. Bring up the stack
docker compose up -d
docker compose ps              # all services should be "running"/"healthy"

# 4. Seed the sources
uv run python generator/generate.py --seed

# 5. Build the platform: ingest -> dbt -> rollups
uv run dagster job execute -m dagster_app -j build_platform

# 6. Open the results
open http://localhost:3000     # Metabase dashboard  (xdg-open on Linux)
open http://localhost:3001     # Dagster UI

Presenting it as a portfolio piece

A project nobody sees can't get you hired. Three moves turn the repo into something an employer can find, trust, and remember.

1. Push it with a clean history. You've been committing per lab — that history is the story of building a platform, and it's a feature. Make sure the public repo has a good description, the README as its landing page, and no secrets in any commit (check that .env was always gitignored; if a secret ever slipped in, rotate it and scrub history). Pin the repo on your GitHub profile.

2. Write a short writeup. A blog post or a LinkedIn post, 300–600 words: what you built, a screenshot of the dashboard and the architecture diagram, and the two or three decisions you're proudest of (why a medallion layout, why you rolled telemetry up hourly, why Dagster over cron). Link the repo. This is what turns a recruiter scroll into a click.

3. Practice a two-minute verbal walkthrough. You will be asked "tell me about a project" in every interview. Rehearse the answer out loud until it's smooth:

• 0:00–0:20 — what & why. "Mini-GridDP is an end-to-end data platform for a GPU-rental marketplace. I built it to practice every layer of a real platform on one machine."
• 0:20–1:20 — the path of one record. Trace a rental from Postgres → bronze → silver (with the SCD2 customer dim) → the gold star schema → a metric → a dashboard chart. The data-flow story shows you understand the whole, not just pieces.
• 1:20–2:00 — a decision and a tradeoff. Pick one thing you chose and one thing you'd do differently at scale (e.g. "DuckDB is perfect on a laptop; in production I'd swap the gold layer onto a warehouse and keep the same dbt models").

A roadmap of extensions

A portfolio piece is more compelling when it's clearly alive — when the README ends with "next steps" and you've actually shipped one or two. Each of these reuses the structure you already have, so the marginal effort is small and the learning is real. Add a short "Roadmap" section to your README listing the ones you plan to do.

Reflect: you built a platform

Step back. Across ten labs you assembled, by hand, every layer of a production data platform — and each piece maps directly to a chapter of the Data Platform Systems Design reference:

Wrapping up the capstone

That's Course 6. You didn't read about a data platform — you built one, end to end, and made it reproducible enough that a stranger can run it. This repo is your proof: concrete, runnable, and yours.

The work isn't to keep building forever. It's to use what you built. A great project sitting in a private repo helps no one — least of all you. The next course is about taking this exact artifact into the job market: Course 7 — Career & Getting the Job turns your platform, your writeup, and your two-minute walkthrough into résumé bullets, interview stories, and offers.

Final commit & tag

Commit the README and any fixes the runbook surfaced, then mark this milestone with a version tag. A tag turns "a repo" into "a release" — a clear, citable point that says this is done and it works.

git add README.md .env.example
git commit -m "Lab 10: README, runbook, and roadmap — platform documented"

# tag the release
git tag -a v1.0 -m "mini-GridDP v1.0 — complete, runnable, documented"

git push origin main
git push origin v1.0

✓ Check yourself

• Does your README open with a one-liner, show the architecture, list the stack, and give one-command run steps?
• Did the run-from-scratch runbook succeed from a fresh /tmp clone — no edits, no hidden state?
• Can you give the two-minute verbal walkthrough out loud, including one decision and one tradeoff?
• Is v1.0 tagged and pushed, with no secrets anywhere in the history?

Next

That's the capstone — and the build. You assembled a complete data platform from the ground up, documented it so anyone can run it, and shipped it as v1.0. This repo is your proof that you can do the job.

Now go get the job. Course 7 — Career & Getting the Job turns this platform into a résumé, an interview story, and an offer — and it's ready now. Congratulations — you've earned this one. 🎉