Building the Data Stack From Scratch

The mandate

Read the posting literally: "establish end-to-end data infrastructure — decide where data sourcing, storage, and querying happens" and "consolidate data access across financial, product, and growth metrics to eliminate duplicate systems." That is not a request for a dashboard. It is a request to own the entire data platform, from the point where a Stripe webhook fires to the point where the VP of Business Operations reads a number and trusts it.

Today that platform does not exist. What exists — at a profitable, roughly-50-person, fast-moving company — is the usual pre-first-hire sprawl: finance has a spreadsheet pulling from the Stripe dashboard, growth has PostHog open in a browser tab, an engineer wrote a one-off script that queries the production Postgres for a board deck, and someone on the token side keeps a Dune dashboard bookmarked. Every one of those is a "system," and several of them disagree about basic facts like how many paying users there are. Your first act of value is not building something new — it is making all of them reconcile to one warehouse and one set of definitions.

Two company-specific constraints shape everything that follows. First, this is a privacy-first company, so your default is self-hosted, OSS tooling that keeps data inside the company's own trust boundary — you should not be shipping product data to a dozen third-party SaaS processors, and you'll want to be the person who says so unprompted (Chapter 02). Second, the product iterates fast, so you build for change: metrics defined once, in code, so a schema churn doesn't silently break the number the CEO quotes on a podcast.

The sources you're consolidating

Before choosing tools, inventory what's actually generating data. Here is the likely source map at the company, and — critically — which sources are content-free (safe to warehouse in full) versus which touch the privacy boundary and must be handled as metadata only. This table is also a great thing to sketch on a whiteboard in the interview: it proves you understand both the business and the constraint.

Note the two genuinely different data-gravity problems here. Stripe, Postgres, PostHog, and Zendesk are standard SaaS/DB sources with off-the-shelf connectors. The API-gateway logs (high-volume, emitted by Rust and Go services) and the on-chain data (a blockchain, not a database) are the two that need custom thought — they're where a naive "just buy Fivetran" answer falls apart.

The reference stack

Here is a defensible end-to-end design. The one-liner to memorize, because it's the whole chapter in a sentence:

Layer by layer, with the choices and the reasoning a senior candidate should be ready to defend:

Warehouse

Start on the existing Postgres — a read replica or a separate analytics schema. At the company's scale (single-digit-millions of users, with API request volume well within a single Postgres's reach), Postgres with good indexing will carry you for a long time and costs nothing new to stand up. When query volume or transformation cost outgrows it, the low-ops moves are BigQuery (serverless, cheap at this scale, trivial to hand out read access) or DuckDB + MotherDuck (astonishingly cheap, self-hostable, great for a small team). Snowflake is overkill early — its cost and ops surface are aimed at a data-team-of-many; adopting it as the first hire is a classic over-engineering tell.

Ingestion

For the standard sources use Airbyte (OSS, self-hosted) — the connectors for Stripe, PostHog, and Zendesk are mature, and self-hosting keeps that data inside the company's boundary, which is the culturally-aligned default here. Fivetran is the managed alternative if you'd rather trade dollars for zero connector maintenance, but sending product data to a third-party processor cuts against the privacy posture and I'd flag that trade-off rather than make it silently. The two custom pieces: a lightweight loader for the API-gateway logs (structured-log ingestion — the Rust/Go services emit JSON lines; ship them to the warehouse via a log pipeline or a small batch job), and an on-chain loader pulling token/credit activity from Dune or Flipside APIs (or an L2 indexer) on a schedule.

Transformation

dbt, non-negotiable. It's where consolidation actually happens: raw sources in, conformed financial / product / growth marts out, with tests and documentation. This is also where metric definitions become code (next section).

BI & experimentation

Self-hosted Metabase for dashboards and self-serve — fast, OSS, easy for non-analysts to slice, and it keeps the BI layer in-boundary. Hex for notebook-style deep dives and narrative analysis. For experiments, PostHog is already chosen (feature flags + A/B) so lean on it now; graduate to GrowthBook (OSS, warehouse-native) if experimentation becomes a core workflow, per Chapter 05.

The whole thing as a data-flow diagram:

  SOURCES                    INGESTION              WAREHOUSE            TRANSFORM              CONSUMPTION
  ---------------------      -----------------      -------------       -----------------      ---------------------
  Stripe (billing)     ─┐                                              ┌ stg_stripe          ┌ Metabase (dashboards,
  Postgres (app DB)    ─┤    Airbyte (OSS,     ─┐                      │ stg_app_db          │   self-serve KPIs)
  PostHog (events)     ─┤    self-hosted)       ├─►  Postgres  ──► dbt ┤ stg_posthog    ──►  ┤
  Zendesk (support)    ─┘                        │   (→ BigQuery /      │                     │ Hex (notebooks,
                                                 │    DuckDB later)     ├ fct_financial       │   deep-dive analysis)
  API-gateway logs     ─── custom log loader  ──►│                      ├ fct_product     ──► │
  (Rust/Go services)                             │                      ├ fct_growth          │ PostHog / GrowthBook
                                                 │                      └ metrics (semantic   │   (experiment tracking)
  On-chain tokens      ─── Dune / Flipside    ──►┘                        layer / marts)      └
  (Ethereum L2)            API loader

  Everything self-hostable ──► data stays inside the company's trust boundary

dbt modeling for consolidation

Consolidation is not "put everything in one database" — it's "define each metric once, in one place, so finance's MRR and growth's MRR are literally the same SQL." dbt gives you a staging → marts layout where raw sources are cleaned in staging, then conformed into marts at a shared grain. The highest-leverage mart for a freemium + API + token business is a customer-day fact that unions billing and usage, because it lets you answer revenue, engagement, and retention questions from one table.

-- One row per paying customer per day: the conformed grain that
-- billing, product, and growth all agree to reconcile against.
-- Sources are already content-free; nothing here touches a prompt.

with billing as (
    select
        customer_id,
        date_day,
        plan_tier,                        -- free / pro / pro_plus / max
        mrr_usd,                          -- normalized to daily-active MRR
        is_active_subscription
    from {{ ref('stg_stripe__subscriptions_daily') }}
),

usage as (                                -- from API-gateway logs, metadata only
    select
        customer_id,
        date_day,
        count(*)                as api_requests,
        sum(tokens_in + tokens_out) as tokens_processed,
        count(distinct model_id)    as models_used,
        avg(latency_ms)             as avg_latency_ms
    from {{ ref('stg_gateway__request_events') }}
    group by 1, 2
),

staking as (                             -- on-chain, cohort-level, consented link only
    select
        customer_id,                     -- populated ONLY where user consented to link wallet
        date_day,
        sum(token_staked)        as token_staked,
        sum(credit_consumed_usd) as credit_consumed_usd
    from {{ ref('stg_onchain__consented_staking_daily') }}
    group by 1, 2
)

select
    b.customer_id,
    b.date_day,
    b.plan_tier,
    b.mrr_usd,
    b.is_active_subscription,
    coalesce(u.api_requests, 0)        as api_requests,
    coalesce(u.tokens_processed, 0)    as tokens_processed,
    coalesce(u.models_used, 0)         as models_used,
    s.token_staked,
    s.credit_consumed_usd
from billing b
left join usage   u using (customer_id, date_day)
left join staking s using (customer_id, date_day)

On top of conformed marts sits the piece that actually kills duplicate systems: a semantic / metrics layer where each business metric is defined once as code. When "active paying subscriber," "MRR," or "NRR" lives in exactly one place, every dashboard and every ad-hoc query inherits the same definition — and a rapid product change means editing one model, not hunting through fourteen dashboards.

# Single source of truth for headline metrics. Change here, every
# dashboard downstream updates. This is the "eliminate duplicate
# systems" mandate made concrete.
metrics:
  - name: mrr
    label: Monthly Recurring Revenue
    model: ref('fct_customer_day')
    calculation_method: sum
    expression: mrr_usd
    filters:
      - field: is_active_subscription
        operator: 'is'
        value: 'true'

  - name: weekly_active_paying_subscribers
    label: Weekly Active Paying Subscribers   # candidate north star (ch. 04)
    model: ref('fct_customer_day')
    calculation_method: count_distinct
    expression: customer_id
    filters:
      - field: plan_tier
        operator: '!='
        value: "'free'"
      - field: api_requests
        operator: '>'
        value: '0'

Build vs buy, and sequencing

A senior first hire is judged as much on what they refuse to build as on what they build. The bias for a small, privacy-conscious, fast-moving team is: buy/adopt the commodity, build only the two things that are genuinely company-shaped.

The trap to name explicitly is over-engineering. The "full modern data stack" — Snowflake + Fivetran + a real-time streaming layer + a data catalog + a separate reverse-ETL tool — over-serves a roughly-50-person company and burns your first quarter on plumbing nobody asked for. The posting itself signals the antidote: it wants someone "comfortable iterating fast rather than perfecting," which is a direct instruction not to gold-plate the infrastructure.

The 30-60-90

The single most likely infrastructure question in the loop is "what would your first 90 days look like?" Here is a plan calibrated to the first-data-hire reality — and be honest about the timeline: at a company with no data foundation, you will not produce a meaningful strategic insight for roughly three months, because the first three months are spent making the data trustworthy. Saying that out loud is a credibility signal, not a weakness.

This is the strategy-and-execution 30-60-90 for the platform. Chapter 10 carries a complementary 30-60-90 framed as the pitch you actually deliver in the room, plus the questions to ask and how to close — read the two together and you'll have both the substance and the delivery. Next, the metrics that platform exists to serve: Chapter 04 · Metrics That Matter.