Start Here

Why a whole course on setup

Most beginners try to skip this part and pay for it later — stuck for hours on an environment problem that has nothing to do with data engineering. Professionals are fast in their environment: they move around the filesystem without thinking, version every change in git, reproduce any setup with a single command. That fluency is a force multiplier for everything that follows.

So we treat the environment as a first-class skill. By the end of this course you'll have the same basic toolkit a working data engineer uses every day, and — more importantly — you'll understand what each piece does.

The toolchain, mapped

Here's everything this course installs and why. Don't install anything yet — just get the lay of the land.

How they fit together — the shape you'll assemble:

A note on your operating system

Data engineering happens on Linux servers, so we work in a Linux-like environment everywhere:

• macOS — already Unix-like; you're good. We'll use the built-in Terminal and install a package manager (Homebrew).
• Linux — you're in the native environment; perfect.
• Windows — install WSL2 (Windows Subsystem for Linux) and do everything inside it. Don't fight the Windows shell for this work; WSL2 gives you a real Ubuntu. Each hands-on chapter notes any Windows-specific step.

How the hands-on chapters work

Starting in Chapter 02, chapters are labs. Each one has a consistent shape so you always know what to do:

If a "You should see" doesn't match what you got, stop and fix it there — debugging the small step now is far easier than discovering it broke three chapters later. (Course 1's "when you get stuck" guidance applies throughout.)

✓ Check yourself

Answer from memory before opening each one.

1. This course installs two databases on your laptop. What role does each one play in the platform you are assembling, and why does a one-person project need both?

   Show answer

   Postgres is a classic relational (OLTP) database and stands in for the source systems that produce data; DuckDB is a fast in-process analytics (OLAP) database and acts as your local warehouse. A pipeline moves data from where it is born to where it is analyzed, so you need one of each end to have anything to build between them. See The toolchain, mapped.

2. In the toolchain map, one tool sits above git, Python and Docker with arrows to all three. Which is it, and what does that position tell you about why it is taught first?

   Show answer

   The shell. Every other tool — and later every server, container and data tool — is driven by typing commands into it, so fluency there multiplies everything that follows; a shaky shell makes each later chapter harder than it needs to be. See The toolchain, mapped.

3. A new teammate spends a day clicking through installers and hand-tuning settings until the pipeline runs on their laptop. It works. By this course's central principle, what is still wrong?

   Show answer

   Nothing about that setup is written down, so nobody — including them, on their next laptop — can rebuild it. The principle is that everything is code and everything is reproducible: environment, pipelines and infrastructure defined in files, kept in git, rebuildable from scratch. A setup that lives only in one person's clicks is "it works on my machine" waiting to happen. See The reproducibility principle.

4. Halfway through a lab, your output doesn't match the "You should see" box, but the next command runs without an error. What do you do, and why?

   Show answer

   Stop and fix it at that step. The expected-output boxes are checkpoints: a mismatch means something upstream is already different, and debugging one small step now is far easier than discovering three chapters later that it broke. "The next command didn't error" is not the same as "my state is right." See How the hands-on chapters work.

5. A friend on Windows is quick in PowerShell and wants to skip WSL2. Another friend already uses git and Docker daily and wants to skip this whole course. What would you tell each of them, and what decides it?

   Show answer

   The Windows friend should install WSL2 anyway: data engineering runs on Linux servers, so the point is to practise in the environment the work actually happens in, not to be fast in a different one. The second friend can reasonably skim — the test is whether they can already move around a shell without thinking, version every change, and rebuild their setup with one command. If so, the labs are a quick check; if not, skipping is what costs hours later on problems that have nothing to do with data. See your operating system and Why a whole course on setup.

uname -s    # prints the OS kernel name
whoami      # prints your username

Next

Tools make more sense once you can see the whole machine they serve. Next we install the mental model — the one picture of how data moves through a platform that the rest of the curriculum hangs on. → The Data Platform Mental Model