Course 8 · Reference

Cheat Sheets

Compact quick-reference cards for the things you look up constantly while building — SQL patterns, the everyday commands, dbt and Dagster, the stack map, and the decisions you'll make again and again.

SQL patterns

The patterns that come up in real work and in interviews. (Depth: Foundations, ch. 5.)

joins & aggregation
-- INNER keeps matches; LEFT keeps all left rows (NULLs where no match)
SELECT c.name, r.gpu_id
FROM customers c
LEFT JOIN rentals r ON r.customer_id = c.customer_id;

-- group + filter groups with HAVING (WHERE filters rows, HAVING filters groups)
SELECT gpu_id, SUM(hours) AS total_hours
FROM rentals
GROUP BY gpu_id
HAVING SUM(hours) > 100;
window functions — the template
SELECT
  customer_id,
  started_at,
  amount,
  -- running total per customer over time
  SUM(amount) OVER (PARTITION BY customer_id ORDER BY started_at) AS running_total,
  -- rank rentals within each customer (1 = most recent)
  ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY started_at DESC) AS rn,
  -- previous row's value
  LAG(amount) OVER (PARTITION BY customer_id ORDER BY started_at) AS prev_amount
FROM rentals;

-- TOP-N PER GROUP: rank, then filter in an outer query
SELECT * FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY gpu_id ORDER BY amount DESC) AS rn
  FROM rentals
) t
WHERE rn <= 3;

-- DEDUP: keep one row per key
SELECT * FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY rental_id ORDER BY loaded_at DESC) AS rn
  FROM bronze_rentals
) t WHERE rn = 1;
CTEs & date bucketing
WITH monthly AS (
  SELECT date_trunc('month', started_at) AS month, SUM(amount) AS revenue
  FROM rentals GROUP BY 1
)
SELECT month, revenue,
       revenue - LAG(revenue) OVER (ORDER BY month) AS mom_change
FROM monthly ORDER BY month;

Shell, git & Docker

The everyday commands. (Depth: Orientation ch. 2–3, Tooling ch. 1–2.)

shell essentials
pwd; ls -la; cd path        # where am I / what's here / move
cat f | grep "err" | wc -l  # pipe: read → filter → count
head -n 100 big.csv > sample.csv   # first 100 rows → new file
find . -name "*.parquet"    # find files by pattern
grep -r "TODO" .            # recursive text search
git — the everyday loop
git status                  # what changed
git switch -c my-feature    # new branch
git add -A && git commit -m "msg"
git push -u origin my-feature   # then open a Pull Request
git switch main && git pull # sync main
git merge my-feature        # (or merge via the PR)
Docker & Compose
docker build -t myimg .     # build an image from a Dockerfile
docker run --rm myimg       # run a container
docker compose up -d        # start the whole stack (detached)
docker compose ps           # what's running
docker compose logs -f svc  # follow a service's logs
docker compose exec postgres psql -U postgres   # shell into a service
docker compose down         # stop it all

dbt & Dagster

(Depth: The DE Craft ch. 7 & ch. 10.)

dbt — a model + test
-- models/marts/fct_revenue.sql
SELECT gpu_id, SUM(hours * price_per_hour) AS revenue
FROM {{ ref('stg_rentals') }}
GROUP BY 1
models/marts/schema.yml
models:
  - name: fct_revenue
    columns:
      - name: gpu_id
        tests: [not_null, unique]
      - name: revenue
        tests:
          - dbt_utils.accepted_range: {min_value: 0}
dbt & Dagster commands / asset
# dbt:  dbt build      (run models + tests)
#       dbt test       (just tests)
#       dbt docs generate && dbt docs serve

# Dagster — an asset with a dependency
from dagster import asset

@asset
def bronze_rentals() -> None:
    ...  # extract from Postgres

@asset(deps=[bronze_rentals])
def fct_revenue() -> None:
    ...  # transform
# run the UI:  dagster dev

The stack at a glance

Each platform layer, the local tool this curriculum teaches it on, and a common production equivalent.

LayerDoesLocal (this curriculum)Common in production
SourceProduces dataPostgres + a generatorApp databases, SaaS, events
IngestMoves data inPython scripts, RedpandaFivetran, Airbyte, Kafka, Debezium
StoreHolds dataDuckDB + Parquet (lakehouse)Snowflake / BigQuery / Iceberg on S3
TransformCleans & modelsdbtdbt, Spark
OrchestrateSchedules & ordersDagsterDagster, Airflow
ServeDelivers to consumersMetabase + a semantic layerLooker, Tableau, semantic layers
Govern / operateQuality, access, monitoringdbt tests, logs, Dagster UICatalogs, observability tools, IAM

Decision heuristics

The recurring "which do I pick?" calls, with a one-line rule of thumb.

DecisionRule of thumb
Batch vs streamingDefault to batch; reach for streaming only when you genuinely need low latency.
Normalize vs dimensionalNormalize OLTP sources (avoid anomalies); model analytics dimensionally (fast, understandable reads).
ETL vs ELTPrefer ELT — land raw, then transform in the warehouse with SQL/dbt.
Warehouse vs lakehouseA warehouse is simpler; choose a lakehouse for open formats, scale, and serving BI + ML from one copy.
Row vs columnar storageRow for transactional point lookups/writes; columnar for analytical scans and aggregation.
SCD Type 1 vs Type 2Type 1 if you don't need history; Type 2 whenever "what was true at the time?" matters.
When to add an indexAdd one when a frequent query filters/joins on a column and reads outweigh writes.
Full vs incremental loadIncremental once the table is large; full loads are fine while data is small.
When to use CDCWhen you need every change (including deletes) and full history, not just the latest snapshot.
View vs table vs incremental (dbt)View for cheap/fresh; table for reused/expensive; incremental for large append-mostly facts.

Rules of thumb & mental models

  • Match the ingest pattern to the source's shape — CDC for mutable DBs, streaming for events, batch for APIs. (Systems Design 1, 4)
  • Declare the grain first — before modeling a fact table, state what one row means. (The DE Craft 2)
  • Never route the firehose through the warehouse — high-volume telemetry goes to a time-series store + rollups. (Systems Design 5, 14)
  • Idempotency makes retries safe — design loads so running twice equals running once. (Foundations 11)
  • The semantic layer is the single source of truth — define each metric once; it's also what makes AI text-to-SQL accurate. (Systems Design 8, 13)
  • Quality is set at the source — the cheapest place to fix data is in the producing system, via contracts. (Systems Design 1)
  • The memory hierarchy spans orders of magnitude — RAM is vastly faster than disk, and the scarce resource; most OOMs are "tried to hold too much at once." (Foundations 8)
  • Everything is code, everything is reproducible — pipelines, config, and infrastructure all in git. (Tooling, all)
  • Build only what's differentiating — buy storage, CDC, and warehouses; build the glue and golden paths. (Systems Design 3)

You've reached the end

That's the whole reference — and the end of the Data Platform curriculum. Eight courses took you from "I can write a basic query and a small script" to a hired, ramping data platform engineer who has designed, built, and operated a real platform.

Keep this glossary and these cheat sheets open as you build; nobody holds it all in their head. When you want the senior, big-picture view of how every piece fits together at scale, the Data Platform Systems Design reference is now yours to read as a peer.

Congratulations

You did the hard part — you kept going. Head back to the curriculum home any time to revisit a course, and go build something real. 🎉