The Data Platform Mental Model
Before you touch a single tool, you need one picture in your head: how a fact about the world travels from the moment it happens to the moment someone — or some model — makes a decision with it. Every tool you install later snaps into a slot in this picture. Learn the picture once and the rest of the curriculum stops feeling like a pile of unrelated technologies and starts feeling like one machine you're assembling.
Data's journey: from "something happened" to "someone decided"
Here is the whole field in one sentence: data engineering is the work of reliably moving data from where it's born to where it's useful, in a shape people can trust. Everything else — the databases, the pipelines, the scheduling tools — exists to make that sentence true at scale.
Picture a single fact. A customer clicks "rent GPU." That click is real for an instant and then gone — unless something captures it, carries it somewhere durable, cleans it up, and presents it to a human or a model. That capture-carry-clean-present trip is the journey, and it has the same handful of stops every time:
Read it left to right. Data is born at a source, ingested (copied) into your platform, parked in storage, transformed from messy raw records into clean, meaningful tables, and finally served to the people and systems that act on it. The arrow that loops back to the top — "someone decided" — is the entire point. A platform that doesn't end in a decision is just an expensive filing cabinet.
If you forget everything else in this course, keep this row of boxes. When you read a job posting, a tool's homepage, or a colleague's design doc, your first question becomes: which box is this about? That question alone will orient you faster than any list of buzzwords.
The platform layers, explained plainly
Those boxes have names — engineers call them layers. Here's each one in plain language, one paragraph apiece.
Sources — where data is born. A source is any system that produces data as a side effect of doing its job: the app database that records orders, the payment processor, a fleet of servers emitting logs, a third-party API, a spreadsheet a human keeps. You usually don't own these — you receive from them. Their defining trait: they were built to run a business, not to be analyzed, so the data comes out shaped for the app, not for you.
Ingestion — moving it in. Ingestion is the act of copying data out of a source and into your platform. Sometimes that's a scheduled job that pulls yesterday's orders every morning; sometimes it's a continuous stream that catches each event as it fires. The hard part isn't the copy — it's doing it reliably, without missing rows or duplicating them, when sources go down or change format underneath you.
Storage — where it lives. Once data is in, it has to rest somewhere durable and cheap enough to keep a lot of it. Two big families live here. A data lake is essentially a giant, cheap folder of files (often raw and semi-structured) — flexible, store-anything. A data warehouse is a database tuned for analytics — structured tables you can query fast with SQL. Most real platforms use both, and the newer lakehouse blends them.
Transformation — cleaning, reshaping, modeling. Raw data is almost never decision-ready. Dates are in five formats, currencies are mixed, the same customer appears under three IDs. Transformation is where you wash and reorganize it: filter junk, join tables together, aggregate millions of events into a tidy "revenue per day" table, and model the business so a number called "active customer" means one agreed thing. This layer is where most of a data engineer's day is actually spent.
Serving — dashboards, APIs, ML. Finally the clean data is handed to whoever consumes it: a business analyst looking at a dashboard, another application calling an API, or a machine-learning model training on the tidy tables you built. Serving is the last mile — if the numbers here are wrong or late, none of the earlier work mattered to the person waiting on them.
Three more concerns don't sit in any single box — they run across all of them. Engineers call these cross-cutting concerns.
Orchestration — scheduling and ordering. Something has to decide what runs, when, and in what order: ingest first, then transform, then refresh the dashboard — and retry the step that failed at 3 a.m. without waking anyone. Orchestration is the conductor that keeps the whole pipeline playing in time.
Data quality — is it right? Moving data fast is worthless if it's wrong. Quality is the set of automated checks that ask, at every stage, "are there null IDs? did row counts suddenly drop? is yesterday's revenue negative?" — and that stop the line before bad data reaches a decision.
Governance — who can see it? Data includes secrets: who bought what, personal details, financials. Governance covers access control (who's allowed to query which tables), privacy, and the documentation that tells people what a column actually means. It's the layer that keeps a powerful platform from becoming a liability.
On a small project one tool might play two layers at once (DuckDB can be both storage and transformation). On a big one, each layer is several tools and a whole team. The layers stay the same regardless — they're how the work is organized, not a shopping list.
A gentle vocabulary primer
A handful of word-pairs come up in every conversation, every doc, every interview. You don't need to memorize them today — just meet them once so they stop being scary. Each gets one plain line:
| Term | In one plain line |
|---|---|
| OLTP vs OLAP | Databases for running the business (many tiny writes) vs analyzing it (big read-heavy queries). |
| Batch vs streaming | Process data in scheduled chunks (e.g. nightly) vs continuously, event-by-event as it arrives. |
| ETL vs ELT | Extract → Transform → Load (clean before storing) vs Extract → Load → Transform (store raw, clean later). |
| Structured / semi-structured / unstructured | Neat rows & columns / loose-but-tagged like JSON / freeform like text, images, audio. |
| Database vs warehouse vs lake vs lakehouse | App's live store / analytics-tuned tables / cheap folder of raw files / a blend of the last two. |
| Pipeline | A sequence of steps that moves and transforms data from a source to a destination. |
| DAG | "Directed Acyclic Graph" — a map of steps and their order, with no loops; how a pipeline is wired. |
| Idempotent | Safe to run twice — running it again produces the same result, not duplicates. A reliability superpower. |
These terms recur in nearly every chapter from here on. You don't have to lock them in now — they'll cement through use. The full Glossary in Course 8 defines every term in the curriculum, so when one slips your mind mid-chapter, that's where to look it up.
One example, traced end to end
Abstractions get real when you push one concrete fact through them. Follow a single GPU rental through all six layers.
Notice what each layer contributed. The source only knew about one rental. Ingestion made a durable copy. Storage kept the raw truth. Transformation turned thousands of individual rentals into one meaningful number per day. Serving put that number in front of a decision. And the whole chain was kept running on time by orchestration, checked for sanity by data quality (did revenue go negative? did row counts crater?), and locked down by governance (the analyst can see daily revenue, but not the customer's payment details). That's the mental model, alive.
Two shapes of data work
The same model stretches to cover two very different rhythms of data, and noticing which you're dealing with shapes every choice downstream.
| Shape | What it looks like | Why it needs different handling |
|---|---|---|
| The firehose | Huge volume of tiny events — clicks, sensor readings, log lines — arriving nonstop. | Sheer volume and speed push you toward streaming and storage built for append-everything-cheaply. |
| Slowly-changing business tables | Modest, important records — customers, products, prices — that change occasionally. | Correctness and history matter more than speed; you track how they change over time, carefully. |
Later courses give each its own techniques — high-throughput ingestion for the firehose, careful modeling for the business tables. For now, just know the model bends to fit both.
How this maps onto the rest of the curriculum
Here's the payoff. Each layer you just met is, roughly, a chunk of the courses ahead. The mental model is your table of contents.
| Layer | Where you'll learn to build it |
|---|---|
| Sources & Storage | Course 3 — databases, the lake/warehouse, how data is modeled and stored. |
| Ingestion | Course 4 — moving data in, batch and streaming, reliably and idempotently. |
| Transformation | Course 5 — cleaning, reshaping, and modeling raw data into trusted tables. |
| Serving + the cross-cutting concerns | Course 6 — dashboards, APIs, and ML feeds, plus orchestration, quality, and governance tying it all together. |
Whenever a later chapter feels disconnected, come back here and ask "which box am I in?" Knowing you're deep in the Transformation layer — and why it sits where it does — turns a wall of new detail into one more room in a house whose floor plan you already know.
✓ Check yourself
- Can you name the five layers in order and say what each one does in a sentence?
- Can you name the three cross-cutting concerns and why they don't fit in a single box?
- Could you explain the difference between a data lake and a data warehouse to a friend?
- Given any tool's homepage, could you guess which layer it serves?
Exercise — Trace an app you use daily through the six layers
Pick an app you open every day. Sketch (on paper or in your head) one thing it produces — a tap, a message, a play, an order — and walk it through: source → ingestion → storage → transformation → serving, then name the decision at the end. Here's a worked answer for a music app.
SOURCE You press play on a song. The app records an event:
play(user=17, track="Heatwave", at=2026-06-21 08:02)
INGESTION Play events stream continuously into the platform as they fire.
(this is a firehose — streaming, not nightly batch)
STORAGE Raw events land cheaply in a data lake, billions of them,
kept exactly as received.
TRANSFORMATION A job aggregates the firehose into tidy tables:
plays_per_track_per_day, and a per-user "top genres" profile.
SERVING (human) An internal dashboard shows which songs are trending this week,
so editors decide what to feature on the home page.
SERVING (ML) A recommendation model reads your genre profile to build
"Made for You" — the decision: which song to queue next.If you could fill in five plausible lines and end with a real decision, the model has landed. Notice you instinctively reached for "streaming, not batch" once you saw the volume — you're already making the kind of judgment call later courses formalize.
Next
You can see the whole machine now. Time to pick up the first tool that drives every layer of it — the shell. → The Command Line