Your Portfolio & Online Presence

Why a portfolio matters more than a degree in data

In a lot of fields, a credential is the ticket. In data engineering, the credential is increasingly the work itself. Hiring managers have been burned by candidates who can recite the difference between a star schema and a snowflake schema but have never actually built a pipeline that runs. So they've learned to discount "trust me" and reward "show me."

That's an enormous advantage for a career-changer. You may not have a CS degree or years on the job, but you can have something a fresh graduate often doesn't: a real, working, documented system that someone can click into and inspect. A portfolio collapses the distance between claiming a skill and demonstrating it.

The capstone (mini-GridDP) as your centerpiece

Your capstone — the mini-GridDP platform — is the single strongest asset in your entire search. Most applicants have toy notebooks; you have an end-to-end system. The job now is presentation: making it trivially easy for a stranger to understand what it does, see it working, and respect the judgment behind it. A strong centerpiece has four layers:

1. A clean public repo. Open-source it on GitHub. Before you do, sweep for embarrassments: no committed API keys or .env files (add them to .gitignore and rotate any that leaked), no 2 GB of raw data, no folder called final_FINAL_v3. A tidy structure signals you've worked on a real team's codebase before, even if you haven't.

2. A great README. This is the most-read file in your portfolio and the one most beginners neglect. The README is the project as far as a skimming recruiter is concerned. It should answer, in order: what is this, what does it do, what does the architecture look like, and how do I run it. Lead with the architecture diagram — a single image that shows data flowing from source → ingestion → warehouse → transformation → dashboard communicates more in three seconds than three paragraphs.

3. A short demo. Code in a repo is abstract; seeing it run is visceral. You don't need a polished video — pick whichever is fastest:

• Screenshots of the dashboard and a successful pipeline run, embedded directly in the README.
• A GIF of the dashboard updating or a DAG turning green — it autoplays in the README and instantly proves "this is real."
• A 2–3 minute Loom walkthrough where you click through the running system and narrate it. This is the single highest-leverage artifact you can make; it's the next best thing to a live demo and recruiters can forward it.

4. A "decisions & tradeoffs" writeup. This is what separates a portfolio that proves a project runs from one that proves you can think. Add a short section (in the README or a linked DECISIONS.md) explaining a few choices: why you used Dagster instead of Airflow, why you partitioned a table the way you did, what breaks at 100× the data volume and how you'd fix it. Naming a limitation honestly is a strength signal, not a weakness — it shows engineering maturity.

Two or three smaller projects that show range

The capstone proves depth. A couple of small, sharp side projects prove range — that you can pick up an unfamiliar tool and ship something focused. Resist the urge to build another giant platform; each of these should be a weekend, not a month, and each should do one thing cleanly. Quality and a good README beat scope.

Writing about your work

A repo shows what you built. A writeup shows how you think — and thinking is what gets hired. A short blog post or project writeup is one of the highest-return things you can produce, because it works while you sleep: it shows up in searches, it gives recruiters something to share, and it gives you a ready-made interview narrative.

A good writeup follows a simple four-beat arc:

• What I built — one or two plain sentences and the architecture diagram.
• Why — the problem it solves, or the skill you wanted to prove.
• What I learned — the surprise, the bug that took a day, the thing the docs didn't tell you. This is the most credible part; specificity reads as real experience.
• What I'd do next — the honest limitations and where you'd take it. This shows you see beyond the demo.

A recruiter-friendly GitHub

Your GitHub profile is, for many roles, the most-clicked link on your resume. A recruiter — often non-technical — spends seconds on it, so optimize for the at-a-glance impression. Here's what they actually look at, and what to do about each:

Here's the kind of README structure that reads well — clean headings, a diagram up top, run instructions, and a decisions section:

# mini-GridDP — an end-to-end energy-grid data platform

A batch data platform that ingests grid sensor readings, loads them into a
warehouse, transforms them with dbt, and serves a metrics dashboard.

![Architecture](docs/architecture.png)

## What it does
- Ingests readings from a public API on a daily schedule (Dagster)
- Lands raw data in DuckDB, then models it staging -> marts with dbt
- Validates row counts and freshness; surfaces metrics in a Streamlit dashboard

## Demo
![Dashboard](docs/dashboard.gif)
▶️ 2-minute walkthrough: https://loom.com/share/...

## Run it locally
```bash
git clone https://github.com/you/mini-griddp.git
cd mini-griddp
cp .env.example .env        # no real secrets committed
make setup && make run      # spins up the pipeline end to end
```

## Decisions & tradeoffs
- **Dagster over Airflow** — asset-based model fit the dependency graph better.
- **DuckDB for the warehouse** — zero-ops and fast for this data size; I'd swap
  in BigQuery/Snowflake past ~50 GB.
- **Known limits** — single-node ingestion; at 100x volume I'd partition by day
  and parallelize the API pulls.

## Tech
Python · SQL · dbt · Dagster · DuckDB · Streamlit · Docker

A LinkedIn recruiters can find

GitHub proves you can do the work; LinkedIn is how recruiters discover you in the first place. Most of them search a keyword database and message whoever matches. Your job is to be the profile that surfaces and reads as credible. Four things move the needle:

• Headline — the single most important line, shown everywhere. Make it a keyword-rich role statement, not a vague aspiration: Data Engineer | Python · SQL · dbt · Airflow/Dagster · BigQuery. Recruiters search those exact terms.
• About — three short paragraphs that tell your story: where you're coming from, the skills you've built (name the tools), and what you're looking for. A career-change arc ("operations analyst who taught myself to build the pipelines I kept wishing existed") is memorable, not a liability.
• Skills & keywords — fill the Skills section with the tools from the postings you're targeting. This section is literally what recruiter searches match against; an empty one makes you invisible.
• Contactable & open — turn on "Open to work" (recruiter-only visibility if you prefer it discreet), add an email or link, and use a real, friendly headshot. An unreachable profile wastes every other bit of effort.

✓ Check yourself

• Could a non-technical recruiter understand what your capstone does from the README alone, in under a minute?
• Does your capstone repo have an architecture diagram, a demo (screenshot/gif/Loom), and a decisions section?
• Are your 3–6 best repos pinned, each with a real README?
• Does your LinkedIn headline contain the exact tool keywords recruiters search for?

# mini-GridDP — an end-to-end energy-grid data platform

mini-GridDP ingests hourly grid-sensor readings from a public API, loads them
into a DuckDB warehouse, and transforms them with dbt into clean marts. A
Dagster schedule runs the whole pipeline nightly and a Streamlit dashboard
serves freshness and load metrics on top.

I built mini-GridDP to prove I could ship a real end-to-end data platform, not
just a notebook. It pulls grid-sensor data from a public API, lands it in
DuckDB, and models it staging-to-marts with dbt, all orchestrated nightly by
Dagster with row-count and freshness checks. The hardest part was making the
load idempotent — my first version double-counted rows on re-runs until I keyed
the merge on (sensor_id, reading_time). I learned that orchestration is mostly
about handling failure and late data gracefully, which the happy-path tutorials
skip entirely. Next I'd partition the warehouse table by day and swap DuckDB for
BigQuery to see how the design holds up past 50 GB.

Next

Your work is now visible and credible. The next step is the document that actually gets submitted — and survives the six-second look and the keyword robot. → The Resume & Applying