Start Here
You can model, ingest, transform, and test data. This course is about everything around that work — the engineering practices and tools that turn a folder of scripts on your laptop into a system a team can run reliably in production. It's the most "software engineering" of the courses, and it's what makes you employable as more than a SQL writer.
What "tooling" means here
"Tooling" isn't a grab-bag of products to memorize. It's the set of practices — and the tools that enable them — for working professionally and at scale with other people:
- Collaborating on a shared codebase without stepping on each other (git workflows, reviews).
- Packaging software so it runs identically everywhere (Docker).
- Assembling many services into one reproducible system (Compose).
- Running on the cloud's primitives (storage, compute, identity).
- Automating testing and deployment (CI/CD).
- Provisioning infrastructure reliably (infrastructure as code).
- Watching it stay healthy (observability).
A pipeline that runs once on your laptop is a demo. A pipeline that runs every night, for years, that a teammate can understand and change, that recovers from failures, and that pages someone when it breaks — that's the job. This course is the gap between the two.
Why the workflow matters as much as the code
Two engineers can write the same transformation. The one whose work is in version control, containerized, tested in CI, deployed by a repeatable process, and monitored is worth far more — because their work is reliable and shareable. Hiring managers screen for exactly this: not "can you write SQL," but "can you ship and operate data systems on a team." Everything in this course is something a real job description lists.
The map of this course
Roughly two halves: chapters 01–03 make your local work professional and reproducible; chapters 04–07 take it toward production — the cloud, automation, and operations.
What it builds on
This course deepens tools you've already met:
| You learned the basics in… | Here we go professional |
|---|---|
| Course 2 — git basics (commit, push) | Ch.01 — branches, pull requests, code review, conflicts |
| Course 2 — running containers | Ch.02–03 — writing Dockerfiles, wiring a multi-service stack |
| Course 4 — the components (dbt, Dagster, etc.) | Ch.03 — assembling them into one orchestrated platform |
| Course 4 — data quality & tests | Ch.05, 07 — running them in CI; monitoring in production |
If any basic feels shaky, revisit the relevant Course 2 chapter first — this course assumes that footing.
Three principles that run through it all
- Everything is code. Application logic, transformations, configuration, and infrastructure all live in version-controlled files. If it's not in git, it doesn't exist.
- Reproducibility. Anyone should be able to recreate your environment and results from scratch with documented commands. No manual snowflakes.
- Automate the repetitive and error-prone. Tests, deploys, and provisioning are done by machines following code, not humans following memory.
✓ Check yourself
Answer from memory before opening each one.
-
A teammate's script loads yesterday's rentals correctly every time they run it from their laptop. By this chapter's standard that is still a demo. What four properties separate it from "the job"?
Show answer
It has to run every night for years without them, be something a teammate can understand and change, recover from failures, and page someone when it breaks. Correct output once, on one machine, proves none of those. See What "tooling" means here.
-
Your
mini-griddprepo now has short-lived branches and reviewed pull requests, a Dockerfile, and one Compose file that brings the whole stack up. Which half of this course does that cover, and what does the other half add?Show answer
That is chapters 01–03: making your local work professional and reproducible. Chapters 04–07 take it toward production — the cloud's primitives (storage, compute, identity, cost), CI/CD, infrastructure as code, and observability. See The map of this course.
-
A colleague creates the storage bucket for GridDP's raw files by clicking through the cloud console, then writes the steps on a wiki page so others can repeat them. Which of the course's three principles does this break, and what would satisfy them?
Show answer
All three, really. The bucket's definition is not in version control ("if it's not in git, it doesn't exist"), it is a manual snowflake nobody can recreate exactly, and provisioning is a human following memory rather than a machine following code. The fix is to define the bucket in version-controlled files and let a tool create it — which is Chapter 06. See Three principles.
-
The data-quality tests you wrote in Course 4 come back twice in this course. What changes about how they are run each time?
Show answer
In Chapter 05 they run in CI, so a machine runs them on every proposed change instead of you remembering to; in Chapter 07 the same idea moves to production, where checks run continuously as monitoring. The tests are the same kind of thing — what changes is that nobody has to remember to run them. See What it builds on.
-
You have one free week before showing
mini-griddpto a hiring manager. You can either write three more sophisticated SQL models, or put the existing ones behind version control, a container, CI and a README a stranger can follow. Which does this chapter argue for, and when would the other choice be right?Show answer
The workflow. Two engineers can write the same transformation; the one whose work is versioned, containerized, tested and repeatable is worth more because it is reliable and shareable, and that is what hiring managers screen for. The other side wins only if the existing models are wrong or too thin to demonstrate anything — reliably shipping incorrect numbers is not an asset. See Why the workflow matters.
Exercise — Audit your own repo (5 minutes)
Open your mini-griddp repo and check honestly: Is it in git with meaningful commits? Is there a README explaining how to run it? Could a stranger reproduce your setup from it? Is there a .gitignore keeping secrets and data out? Jot down what's missing.
There's no single solution — this is a self-audit. By the end of this course, every one of those gaps will be closed, and your repo will look like a professional project. That transformation is, in a real sense, the whole point of the course.
Next
We start with the skill you'll use more than any other on a team: collaborating in a shared git repository without chaos. → Professional Git & Collaboration