Start Here
You can model, ingest, transform, and test data. This course is about everything around that work — the engineering practices and tools that turn a folder of scripts on your laptop into a system a team can run reliably in production. It's the most "software engineering" of the courses, and it's what makes you employable as more than a SQL writer.
What "tooling" means here
"Tooling" isn't a grab-bag of products to memorize. It's the set of practices — and the tools that enable them — for working professionally and at scale with other people:
- Collaborating on a shared codebase without stepping on each other (git workflows, reviews).
- Packaging software so it runs identically everywhere (Docker).
- Assembling many services into one reproducible system (Compose).
- Running on the cloud's primitives (storage, compute, identity).
- Automating testing and deployment (CI/CD).
- Provisioning infrastructure reliably (infrastructure as code).
- Watching it stay healthy (observability).
A pipeline that runs once on your laptop is a demo. A pipeline that runs every night, for years, that a teammate can understand and change, that recovers from failures, and that pages someone when it breaks — that's the job. This course is the gap between the two.
Why the workflow matters as much as the code
Two engineers can write the same transformation. The one whose work is in version control, containerized, tested in CI, deployed by a repeatable process, and monitored is worth far more — because their work is reliable and shareable. Hiring managers screen for exactly this: not "can you write SQL," but "can you ship and operate data systems on a team." Everything in this course is something a real job description lists.
The map of this course
Roughly two halves: chapters 01–03 make your local work professional and reproducible; chapters 04–07 take it toward production — the cloud, automation, and operations.
What it builds on
This course deepens tools you've already met:
| You learned the basics in… | Here we go professional |
|---|---|
| Course 2 — git basics (commit, push) | Ch.01 — branches, pull requests, code review, conflicts |
| Course 2 — running containers | Ch.02–03 — writing Dockerfiles, wiring a multi-service stack |
| Course 4 — the components (dbt, Dagster, etc.) | Ch.03 — assembling them into one orchestrated platform |
| Course 4 — data quality & tests | Ch.05, 07 — running them in CI; monitoring in production |
If any basic feels shaky, revisit the relevant Course 2 chapter first — this course assumes that footing.
Three principles that run through it all
- Everything is code. Application logic, transformations, configuration, and infrastructure all live in version-controlled files. If it's not in git, it doesn't exist.
- Reproducibility. Anyone should be able to recreate your environment and results from scratch with documented commands. No manual snowflakes.
- Automate the repetitive and error-prone. Tests, deploys, and provisioning are done by machines following code, not humans following memory.
✓ Check yourself
- Can you say, in one sentence, the difference between a pipeline that "works on my machine" and one that's production-ready?
- Can you name the seven practices that make up "tooling"?
- Are your Course 2 git and Docker basics solid? (You'll build directly on them.)
Exercise — Audit your own repo (5 minutes)
Open your mini-griddp repo and check honestly: Is it in git with meaningful commits? Is there a README explaining how to run it? Could a stranger reproduce your setup from it? Is there a .gitignore keeping secrets and data out? Jot down what's missing.
There's no single solution — this is a self-audit. By the end of this course, every one of those gaps will be closed, and your repo will look like a professional project. That transformation is, in a real sense, the whole point of the course.
Next
We start with the skill you'll use more than any other on a team: collaborating in a shared git repository without chaos. → Professional Git & Collaboration