Lab 08 — Serve It — BI Dashboard

The goal

Every lab so far has moved data toward a destination it never quite reached: a human making a decision. The generator produced events, ingestion landed them, dbt modeled them into a star schema, the rollup tamed the telemetry firehose, orchestration ran it on a schedule, tests guarded it, and Lab 07 defined the metrics. Now we close the loop. We're going to plug a business intelligence (BI) tool — Metabase — into the warehouse and build a dashboard for the imaginary people who run our GPU marketplace: the ops lead who cares about utilization, the finance lead who cares about GMV, the account manager who cares about top customers.

Here's the slice of the architecture this lab completes — the far-right "SERVE" column:

By the end you'll have a single dashboard with five charts — GMV over time, utilization by GPU model, ARPU per month, top customers by revenue, and telemetry health — all reading from the governed gold layer so the numbers match everywhere.

Connect Metabase to the warehouse

This is the one genuinely fiddly part of the lab, so let's be honest about it up front. Our warehouse is DuckDB — a fantastic embedded analytics engine, but an awkward fit for a server BI tool. Metabase has no built-in DuckDB driver; there's a community driver you can drop in, but it lags Metabase releases, needs a matching .jar version, and breaks when DuckDB has the file open from another process (DuckDB allows only one writer). On a laptop that means flaky "database is locked" errors right when you're demoing.

So you have two paths:

Step 1 — Publish gold to Postgres

We already have Postgres running (it's our source for rentals/customers, but a separate schema keeps serving data clean). The simplest robust approach: a tiny publish script that reads each gold table from DuckDB and writes it into a serving schema in Postgres. DuckDB can attach Postgres directly, so this is a handful of lines:

"""Publish gold marts + metric views from DuckDB into a Postgres `serving` schema
so Metabase (which has no DuckDB driver) can read them natively."""
import duckdb, os

PG = os.environ.get(
    "SERVING_PG_DSN",
    "postgresql://griddp:griddp@localhost:5432/marketplace",   # the Lab 01 database
)
DUCKDB_PATH = os.environ.get("DUCKDB_PATH", "warehouse.duckdb")

# Tables to serve: the gold star schema + the Lab 07 metric views.
TABLES = [
    ("gold", "dim_customer"), ("gold", "dim_gpu"), ("gold", "dim_date"),
    ("gold", "fct_rentals"), ("gold", "fct_telemetry_hourly"),
    ("metrics", "metric_revenue"), ("metrics", "metric_arpu"),
    ("metrics", "metric_revenue_by_model"), ("metrics", "metric_utilization"),
]

# Not read_only=True: DuckDB would then open the attached Postgres read-only too.
con = duckdb.connect(DUCKDB_PATH)
con.execute("INSTALL postgres; LOAD postgres;")
con.execute(f"ATTACH '{PG}' AS pg (TYPE postgres);")
con.execute("CREATE SCHEMA IF NOT EXISTS pg.serving;")

for schema, t in TABLES:
    print(f"publishing {schema}.{t} -> serving.{t}")
    con.execute(f"DROP TABLE IF EXISTS pg.serving.{t};")
    con.execute(f"CREATE TABLE pg.serving.{t} AS SELECT * FROM {schema}.{t};")

print("done — gold published to Postgres serving schema")
con.close()

The defaults match the Lab 01 database and the gold and metrics schemas from Labs 03 and 07; set SERVING_PG_DSN only if yours differ. The views become plain tables on the Postgres side, which is what you want a BI tool to hit. Run it from the repo root:

uv run python ingest/publish_to_serving.py

Step 2 — Point Metabase at the serving database

Open Metabase at http://localhost:3001 (Compose publishes it on 3001; 3000 is Dagster). On first run it walks you through creating an admin account (use any email/password — it's local). Then add the database:

Build the dashboard

Metabase is a GUI, so there are no files to type here — but every chart is just a saved question, and behind every question is SQL. We'll give you the SQL for each so you understand exactly what the chart shows (and so you can paste it into Metabase's native SQL editor, which is the fastest path). For each: click + New → SQL query, pick the serving database, paste the SQL, run, choose the visualization, and Save. Then create a dashboard called "Marketplace Overview" and add each saved question to it.

1 — GMV over time (line chart)

GMV (gross merchandise value) is the headline number — total rental value billed per day. Read it straight from the governed metric_revenue you published so it matches finance's number exactly.

SELECT rental_date, gmv
FROM serving.metric_revenue
ORDER BY rental_date;

Visualization: Line, x = rental_date, y = gmv. Format the y-axis as currency.

2 — Utilization % by GPU model (bar chart)

The ops lead's number: how hard each GPU model is working. It comes from the Lab 07 metric_utilization, which already weights every hour of the telemetry rollup by its sample count, so the chart cannot quietly average the averages.

SELECT model, utilization_pct
FROM serving.metric_utilization
ORDER BY utilization_pct DESC;

Visualization: Bar, x = model, y = utilization_pct. With the Lab 04 producer every model sits in the mid-70s, so the bars are close; a real fleet would not be.

3 — ARPU per month (line / single number)

The finance lead's second number: average take-rate revenue per active customer, per month, straight from metric_arpu. It's a health metric for the customer base: falling ARPU means more customers are spending less.

SELECT year_month, arpu, active_customers
FROM serving.metric_arpu
ORDER BY year_month;

Visualization: Line for the trend, or a Number showing the latest full month. Format arpu as currency. The current month is partial, so its value is low; that is the data, not a bug.

4 — Top customers by revenue (table or bar)

The account manager's view: who are our biggest customers? Join the rentals fact to the customer dimension and sum revenue.

SELECT c.customer_name,
       c.tier,
       ROUND(SUM(r.revenue), 2) AS total_revenue,
       COUNT(*)                 AS rentals
FROM serving.fct_rentals r
JOIN serving.dim_customer c ON c.customer_id = r.customer_id
GROUP BY c.customer_name, c.tier
ORDER BY total_revenue DESC
LIMIT 10;

Visualization: Table (clean for a top-10), or a horizontal Bar with customer_name on the y-axis.

5 — Telemetry health (single number)

A platform-trust metric: are the GPUs actually reporting? If telemetry goes quiet, every utilization number above is suspect. We count how many GPUs reported in the most recent hour versus how many exist.

WITH latest AS (
  SELECT MAX(hour) AS h FROM serving.fct_telemetry_hourly
)
SELECT COUNT(DISTINCT t.gpu_id)             AS reporting_gpus,
       (SELECT COUNT(*) FROM serving.dim_gpu) AS total_gpus
FROM serving.fct_telemetry_hourly t, latest
WHERE t.hour = latest.h;

Visualization: Number (or two numbers side by side). When reporting_gpus drops below total_gpus, something upstream is dropping telemetry — exactly the kind of "is my data even arriving?" check Lab 09 will formalize.

Build on the governed layer

Look back at those five queries. Notice what they don't do: they never recompute a business definition in the BI tool. GMV, utilization and ARPU come straight from the serving.metric_* tables — the metrics you defined once in Lab 07. Top customers and telemetry health read the gold facts and dimensions, not raw bronze. That discipline is the whole point of a governed dashboard.

The failure mode it prevents is the classic one: finance's slide says GMV was $4.1M, the ops dashboard says $3.8M, and a meeting gets eaten arguing about whose number is right. The cause is almost always that someone redefined "GMV" with a slightly different filter inside a BI question. When every chart reads the same marts and the same metrics table, that argument can't happen — the numbers agree because they come from one definition.

Run it & view the dashboard

Let's do a clean end-to-end pass, the way you would before showing it to someone. From the repo root, with the stack up:

# 1. rebuild gold + metrics so the warehouse is current
cd dbt && uv run dbt build && cd ..

# 2. publish the fresh gold into the Postgres serving schema
uv run python ingest/publish_to_serving.py

# 3. (in Metabase) hit the refresh icon on the dashboard, or
#    Settings -> Databases -> serving -> "Sync database schema now"

Open http://localhost:3001 and go to your Marketplace Overview dashboard.

Commit Lab 08

Two things changed: the serving step (the publish script) and your dashboard's definitions. Metabase stores dashboards in its own database, but you should still commit the SQL and notes so the dashboard is reproducible from the repo — drop them in the dashboards/ folder the skeleton reserved.

# save each question's SQL + a short README under dashboards/
mkdir -p dashboards/marketplace-overview
# (paste each query into dashboards/marketplace-overview/*.sql,
#  and note the chart type + dashboard layout in a README.md)

git add ingest/publish_to_serving.py dashboards/
git commit -m "Lab 08: serve gold to Postgres + Marketplace Overview dashboard"

✓ Check yourself

Answer from memory before opening each one.

1. You point Metabase straight at warehouse.duckdb with the community driver. Mid-demo, the scheduled Dagster run kicks off. What is the likely failure, and why does Path B not have it?

   Show answer

   "Database is locked". DuckDB allows one writer on the file, and now dbt/Dagster and Metabase are contending for it (on top of the driver's version sensitivity). Path B publishes gold into Postgres, which serves concurrent readers natively, so the BI tool never competes with the pipeline for a file lock — compute and serving are separated. See Connect Metabase to the warehouse.

2. You fix a bug in fct_rentals, run dbt build, and the dashboard still shows the old numbers. Why — and where should the missing step live so this cannot happen?

   Show answer

   dbt rebuilt gold in DuckDB, but Metabase reads the copy in the Postgres serving schema, and that only changes when publish_to_serving.py runs. It belongs in Dagster as the last asset in the graph, downstream of gold, so a fresh build always ends with a fresh serving copy. See This belongs in Dagster and Run it.

3. The publish script does DROP TABLE and then CREATE TABLE … AS for each table. A stakeholder refreshes the dashboard while fct_rentals is between the two. What do they see, and how would you change the publish to avoid it?

   Show answer

   An error: for that moment serving.fct_rentals does not exist, and the moment grows with the table. Build the new copy under a temporary name and swap it in — a rename, or the drop and create inside one transaction — so a reader always finds a complete table. Drop-and-recreate is idempotent, but it is not invisible to readers. See Step 1 — Publish gold to Postgres.

4. The telemetry-health tile reads reporting_gpus = 7, total_gpus = 10. Which other tile do you stop trusting, which are unaffected, and what do you check before raising an alarm?

   Show answer

   Utilization by GPU model — it is derived from the telemetry rollup, so three silent cards skew it. GMV, ARPU and top customers come from rentals and the metric tables and are untouched. Before alarming anyone, check whether those three cards simply had no active rental in the latest hour (telemetry is emitted per active rental) or whether readings are being dropped upstream. See Build the dashboard.

5. The account manager wants "GMV for enterprise customers only". metric_revenue has no tier column, and you could write the number in a chart from fct_rentals in five minutes. What decides whether you should?

   Show answer

   Whether the number will outlive today. A chart-level query re-derives GMV with its own filter and expression, which is exactly how the $4.1M-versus-$3.8M argument starts. For throwaway exploration, labelled as such, that is fine. If it will sit on the dashboard beside the governed GMV, or be reused, define it upstream — GMV by tier in dbt, tested and published — so it cannot drift. See Build on the governed layer.

SELECT c.tier,
       ROUND(SUM(r.revenue), 2) AS total_revenue,
       COUNT(DISTINCT c.customer_id) AS customers
FROM serving.fct_rentals r
JOIN serving.dim_customer c ON c.customer_id = r.customer_id
GROUP BY c.tier
ORDER BY total_revenue DESC;

git add dashboards/marketplace-overview/revenue_by_tier.sql
git commit -m "Lab 08: add revenue-by-tier chart"

Takeaway

• Metabase has no built-in DuckDB driver, and the community driver is version-sensitive and fights Dagster and dbt for the file lock, so a small publish step copies the gold and metric tables into a Postgres serving schema.
• Separating the analytics engine from the serving store is the standard pattern, because the store BI reads should be a stable copy that does not compete with pipelines for a lock.
• Every chart reads a published metric table instead of recomputing GMV, utilization or ARPU, so the dashboard cannot disagree with finance.
• Build charts only on tested gold marts and never on bronze, and let shared definitions such as active customer live upstream.
• Empty charts almost always mean unpublished or stale data, so re-run the publish script and sync the schema before you debug the SQL.
• Metabase keeps dashboards in its own database, so commit each chart's SQL and you can rebuild the dashboard if the volume is wiped.
• A tile that compares reporting GPUs with total GPUs makes silent telemetry loss visible before it corrupts every utilization number.

The platform now serves answers, but a build that only works on your laptop is not shippable, so the next lab adds automated checks and run visibility. → Lab 09 — CI/CD & Observability

Further reading

• Metabase: Adding and managing databases — Metabase docs, accessed 2026. Connecting Metabase to the Postgres serving schema.
• Metabase: Questions — Metabase docs, accessed 2026. Saved questions and the SQL behind them.
• Metabase: Introduction to dashboards — Metabase docs, accessed 2026. Assembling questions into one dashboard.
• DuckDB: PostgreSQL extension — DuckDB docs, accessed 2026. Writing DuckDB tables into Postgres, the basis of the publish step.
• DuckDB: Concurrency — DuckDB docs, accessed 2026. The single-writer rule that makes a separate serving store necessary.