The Application Space

Why a DPE starts at the source, not the warehouse

The instinct of a new data engineer is to start at the warehouse: pick Snowflake, write some dbt, build a dashboard. The instinct of a senior DPE is to spend the first weeks mapping the application space — because the source systems impose constraints that no amount of downstream cleverness can undo. If a source overwrites rows in place with no change-data-capture, you cannot reconstruct history. If an event stream has no idempotency key, you cannot dedupe. If telemetry arrives without event-time timestamps, every time-window aggregation is wrong.

So we begin with the three structural categories from Start Here: transactional systems, event streams, and third-party systems. Each maps to a different ingestion pattern (chapter 04), and the split is the single most useful lens for the whole platform.

Transactional systems (OLTP)

These are the application databases that run the product. They are row-oriented, normalized, mutable, and authoritative — the system of record for the business state. GridDP has four.

1 · marketplace-api (Postgres)

The heart of the product: accounts, listings/offers, search configuration, rentals, and bids. Highly normalized, heavily indexed for point lookups, updated thousands of times per second.

• Key tables: account, host, machine, gpu, offer, rental, bid.
• Mutation pattern: rows update in place — an offer.price changes, a rental.status moves pending → active → stopped. This is the central challenge: without capturing those transitions, the warehouse only ever sees the latest state.
• Ingest implication: needs log-based CDC (Debezium reading the Postgres WAL) to stream every insert/update/delete as an event. We design this in chapter 04 and turn it into history with slowly-changing dimensions in chapter 06.

2 · billing-service (Postgres)

Metering, invoices, credits, payouts, and a double-entry ledger. The correctness bar here is the highest in the company: this data reconciles to money.

• Key tables: rental_meter (the per-rental-hour usage atom), invoice, credit, payout, ledger_entry.
• Correctness need: exactly-once semantics end-to-end. A double-counted meter row is a double charge. The ledger must always balance (debits = credits) — a property we assert as a data test in chapter 10.
• Ingest implication: CDC again, but with stricter reconciliation. Finance will want the warehouse total to tie to the ledger to the penny (chapter 08, 11).

3 · instance-orchestrator (Postgres + emits events)

Manages the lifecycle of a running instance on a host: scheduling a container, starting it, stopping it, handling a preemption when a higher interruptible bid arrives. It is half OLTP (current instance state) and half event producer (lifecycle transitions).

• Key tables/events: instance (state), and lifecycle events instance.created/started/stopped/preempted/destroyed.
• Why it matters: the join between an instance and its telemetry_sample stream is how we attribute physical GPU behavior to a specific paying rental. Get the lifecycle timestamps wrong and utilization, billing, and reliability all drift.

4 · ranking-service (Postgres + recomputed)

Stores the trust signals that the marketplace surfaces in search: DLPerf benchmark scores, reliability scores, and the derived search-ranking features. Interesting because it is both a source and a consumer — it reads telemetry-derived features the platform computes and writes back scores the product shows. This feedback loop is why the data platform sits in the product's critical path, not just the back office.

Event streams

Where OLTP captures state, event streams capture things that happened — append-only, immutable, high-volume, ordered (within a partition). GridDP publishes these to Kafka topics.

6 · marketplace-events

The behavioral and market stream: search.performed, offer.viewed, rental.requested, bid.placed, bid.outbid, instance.preempted, rental.cancelled, plus web/app clickstream. This is what powers funnels, liquidity analysis, and the bid-market reconstruction.

• Shape: JSON or Avro/Protobuf messages with an event_id, event_time, actor_id, and a typed payload. The presence of event_id (for dedupe) and event_time (for correct windowing) is non-negotiable — chapter 04 explains why.
• Volume: bursty. A popular open-model release triggers a fine-tuning rush and the bid stream spikes 10×. The platform must absorb spikes without backpressure into the product.

The telemetry firehose

This source deserves its own section because it dominates the platform. Every GPU on every machine runs a host-agent that reports physical metrics on a tight interval.

7 · host-agent telemetry ⚡

What makes the firehose hard is the combination of properties:

• Volume: ~150 M rows/day, growing linearly with fleet size. A naïve load into a columnar warehouse is both slow and expensive.
• Time-series access pattern: almost every query is "metric X for entity Y over time window W" — which a purpose-built engine (ClickHouse) serves 10–100× more cheaply than a general warehouse.
• Lateness & gaps: a host with flaky residential internet (picture residential, hobbyist-grade providers) drops offline and backfills. Gaps are themselves signal — they feed the reliability score.
• Dual use: it powers operational dashboards (is the fleet healthy now?) and ML features (predict reliability, detect throttling) and billing inputs (was the GPU actually delivering during the rented hour?). One source, three very different SLAs.

Third-party systems

Data the company does not produce but must integrate. These are pulled, not pushed, and carry their own rate limits, schemas, and compliance weight.

8 · payments — Stripe, crypto rails, KYC/AML

• Stripe: charges, refunds, disputes, payout transfers. Ingested via webhooks (events) plus a nightly API reconciliation pull. The webhook/API pair is classic: webhooks are timely but can be missed; the batch pull is the source of truth.
• Crypto rails: on-chain deposits/withdrawals (picture crypto-native providers). Confirmation latency and reorgs make "is this payment final?" a genuine data-modeling question.
• KYC/AML provider: identity verification outcomes. PII-heavy — drives the governance design in chapter 11 (tokenization, restricted domains, audited access).

9 · support — Zendesk-like

Tickets, disputes, CSAT. Lower volume, but joins to the customer and trust-and-safety domains to answer questions like "do reliability complaints predict churn?" Ingested by batch API pull (Fivetran-style connector in Stack C; a scheduled extract in Stack A).

Data shapes & ingest patterns — the master map

Pulling it together, here is the decision table that drives chapter 04. Match the ingest pattern to the source's shape, not to your favorite tool.

Three ingest patterns cover all nine sources: log-based CDC for mutable OLTP, streaming for append-only events, and batch API pull for third-party systems. The firehose is "streaming, but routed to a specialist store." That's the whole ingestion design in one sentence — chapter 04 fills in the mechanics.

Source ownership & data contracts

Sources are owned by product engineering teams who did not build them for analytics. The schema can change without warning the moment a feature ships — a renamed column, a new enum value, a dropped field — and your pipeline breaks at 3 a.m. The senior-level answer is the data contract: an explicit, versioned agreement between the producing team and the platform about the schema, semantics, and SLAs of what a source emits.

contract: bid.placed
owner: marketplace-team
version: 2.1.0
schema:
  event_id:   { type: uuid,      required: true,  unique: true }   # dedupe key
  event_time: { type: timestamp, required: true }                  # producer-stamped, UTC
  account_id: { type: string,    required: true }
  offer_id:   { type: string,    required: true }
  bid_usd_hr: { type: decimal,   required: true,  min: 0 }
  is_interruptible: { type: bool, required: true }
guarantees:
  freshness_p99: 60s          # platform may assume data lands within 60s
  ordering: per-account_id    # ordered within a partition key
  pii: false
breaking_change_policy: major-version-bump + 2-week notice

Contracts move data quality left — to the source, where it's cheapest to fix. They're how a platform scales past the point where the DPE can personally know every schema. We operationalize them (tests, CI checks, registry) in chapters 09 and 10.

Source anti-patterns to catch early

Takeaway

• A DPE maps the application space before touching the warehouse, because sources impose constraints downstream work cannot undo.
• GridDP's nine sources fall into three categories — OLTP (CDC), events (streaming), third-party (batch) — and the category, not the tool, dictates the ingest pattern.
• One source, the telemetry firehose, dominates volume and cost; treating it like the others is the signature scaling mistake.
• Data contracts push quality to the source and let the platform scale past tribal knowledge.

Next we zoom out from the data to the role: what a data platform actually is, how the DPE differs from adjacent roles, and the stakeholder map the platform must serve. → The Data Platform & the DPE Role