Build vs Buy & the Platform Landscape

Why this is the decision that recurs

Batch vs streaming and lake vs warehouse each fire once or twice in a design. Build vs buy fires at every layer — ingestion, storage, transform, orchestration, catalog, quality, BI. You'll make this call seven times in one session, and interviewers use it to probe the thing they can't test any other way: do you understand what a data platform actually costs to own, and can you tell what's worth your team's scarce engineering time?

The trap is treating it as a binary or a matter of taste. "We'd build it, we're a strong team" and "just buy Fivetran, everyone does" are both junior answers — one ignores opportunity cost, the other ignores that a $40k/month Fivetran bill at high connector volume can justify a quarter of engineering time. The senior answer is a repeatable framework applied per component, always tied back to the requirements from Chapter 02.

The build-vs-buy framework

Run every layer through the same eight lenses. In the room, you won't recite all eight — you'll pick the two or three that dominate for that component and name them out loud.

• Total cost of ownership (TCO), not license price. The invoice is the visible tip. Add the headcount to build and operate it, the ops burden (on-call, upgrades, capacity planning), and the opportunity cost of what those engineers aren't building. A "free" self-hosted Airflow cluster that consumes one engineer's attention costs more than a managed orchestrator's bill.
• Time-to-value. Buy gets you a working ingestion path this afternoon; build is a quarter to first production data, plus a maintenance tail. Early-stage or unproven use cases favor buy hard — you don't build infrastructure for a hypothesis.
• Differentiation — build core, buy commodity. The single most important lens. Does this component create competitive advantage? Loading Salesforce into a warehouse is a solved commodity — buy it. A recommendation feature-computation layer that is your product might be worth building. Never spend build capacity on undifferentiated heavy lifting.
• Team size & maturity. A five-person data team can't operate self-hosted Kafka, Airflow, Trino, and a catalog and still ship. Buy-heavy stacks exist precisely so small teams punch above their weight. A 50-engineer platform org has the muscle to run OSS and capture the savings.
• Reliability / SLA needs. A vendor with a 99.9% SLA and a 24/7 SRE team may beat what your team can operate — or a strict internal SLA on a differentiating path may demand control a vendor won't give you.
• Vendor lock-in & exit cost. How hard is it to leave? Proprietary storage formats, SQL dialects, and closed metadata raise exit cost. Open formats (Iceberg/Delta, Parquet) and standard interfaces keep the door open. Ask: if this vendor triples the price, what's my BATNA?
• Data gravity & egress. Data is heavy and expensive to move. Cloud egress fees and the sheer time to move petabytes create gravity — tools that live next to your data (warehouse-native transforms, in-cloud engines) win, and cross-cloud SaaS can incur egress that dwarfs the license.
• Security & compliance. Can the data leave your VPC? Regulated data (PII, PHI, financial) may forbid a multi-tenant SaaS, mandate a customer-managed key, a private deployment, or in-region processing — sometimes ruling out buy regardless of cost.

The spectrum: SaaS to in-house

Build vs buy isn't binary — it's a spectrum of how much operational responsibility you keep. Naming the middle points shows you know the real options.

   BUY  ◄─────────────────────────────────────────────────────────────►  BUILD
        less control, less ops                          more control, more ops
        faster time-to-value                            higher fixed cost

  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐   ┌──────────────┐
  │ Managed SaaS │   │ Managed OSS  │   │Self-hosted OSS│  │  In-house    │
  ├──────────────┤   ├──────────────┤   ├──────────────┤   ├──────────────┤
  │ Fivetran     │   │ Confluent    │   │ Kafka on k8s │   │ Custom CDC   │
  │ Snowflake    │   │  Cloud       │   │ Airflow OSS  │   │ in-house     │
  │ BigQuery     │   │ Databricks   │   │ Trino        │   │ orchestrator │
  │ dbt Cloud    │   │ MWAA / Astro │   │ Iceberg+     │   │ bespoke      │
  │ Looker       │   │ (Astronomer) │   │  Trino self  │   │ storage svc  │
  └──────────────┘   └──────────────┘   └──────────────┘   └──────────────┘
     vendor runs        vendor runs        you run the        you write AND
     everything         the OSS for you    OSS yourself       run the code

Managed OSS is the underrated middle. Confluent Cloud (Kafka), Databricks (Spark), Amazon MWAA / Astronomer (Airflow), and Snowflake-on-Iceberg let you run open, portable technology while a vendor carries the ops. You get most of the exit-cost protection of OSS with most of the low-ops of SaaS — often the right answer for a mid-size team that wants portability without a platform team. Don't jump straight from "managed SaaS" to "self-host it ourselves."

Modern data stack vs consolidated

Two archetypes dominate real designs, and naming them frames the whole conversation.

The modern data stack (MDS) is deliberately buy-heavy and best-of-breed: Fivetran/Airbyte for ingestion, Snowflake/BigQuery for storage & compute, dbt for transformation, a BI tool (Looker/Tableau/Mode), and reverse ETL (Census/Hightouch) to push modeled data back into operational tools. Each layer is a specialist SaaS, glued by SQL and orchestration. It's the fastest way for a small team to a working platform, and it's what most startups run.

The consolidated / OSS archetype trades integration surface for control and cost. Either a single platform absorbs many layers (Databricks doing ingestion + storage + transform + serving; or a warehouse-native stack), or a self-hosted OSS estate (Kafka + Airflow + Spark/Trino + Iceberg + a catalog like Nessie or Polaris) runs on infrastructure you own. Fewer vendors, fewer contracts, lower marginal cost at scale — but a real platform team to run it.

Component-by-component: typical build vs buy

The framework resolves differently per layer because differentiation and commodity-ness differ per layer. This table is the artifact to reproduce on the whiteboard when an interviewer asks "so what would you buy and what would you build?"

Read the "typical default" column top to bottom: the platform is mostly buy, with build reserved for the layers where scale economics or genuine differentiation appear. Note also that transformation and quality almost never get built from zero — dbt Core and Great Expectations are free, mature floors, so "we'd build our own transform framework" should raise your own eyebrow before the interviewer's.

When building is actually right

Buy is the default, but there are three durable reasons build wins. Name which one applies — a build decision without one of these is usually a mistake.

• Scale economics. Consumption-priced SaaS is cheap until it isn't. At extreme volume the vendor bill grows linearly with rows/queries while a self-hosted alternative's cost is dominated by fixed infrastructure + a stable team. The classic example: companies that outgrew Fivetran's connector pricing and moved high-volume sources to self-hosted Airbyte or Kafka Connect, or moved off a per-TB-scanned warehouse to Iceberg + Trino. The test: does the annual bill exceed the fully-loaded cost of the team required to run the alternative, with margin?
• Unique requirements. No vendor fits — a source with no connector, a latency requirement no managed engine meets, a compliance constraint (data can't leave your VPC, in-region only, customer-managed keys) that rules out multi-tenant SaaS, or a transformation that is the product. This is the "differentiation" lens saying yes.
• Cost at extreme volume, specifically. A special case of scale economics worth calling out because it's the most common real trigger: at petabyte storage or tens-of-billions-of-events/day, the marginal cost of a managed warehouse or a per-row connector dominates everything, and open storage on object stores (Parquet/Iceberg on S3) with an engine you scale yourself becomes dramatically cheaper — even after paying the platform team.

The decision scorecard

Two artifacts win this discussion. First, the qualitative decision matrix — the axes you weigh. Second, a lightweight scorecard you can run per component on the whiteboard when the interviewer wants rigor.

To make it decisive, score each factor 1–5 for how strongly it pulls, weight the two or three that dominate for this component, and add them up. The point isn't false precision — it's forcing yourself to name which factors carry weight here, which is exactly what a defended trade-off requires.

The heuristic & how to defend it

The corollaries that make it operational:

• Buy the commodity, build the core. If the layer isn't a competitive advantage, it's a candidate for buy. Save build capacity for the thing only your company would ever make.
• Managed OSS is the hedge. When you want portability but not the ops, sit in the middle — Confluent, Databricks, Astronomer, Snowflake-on-Iceberg.
• Match the stack to the team. A five-person team runs a buy-heavy modern data stack; a platform org can capture OSS savings. Recommending self-hosted Kafka to a tiny team is a red flag.
• Design against lock-in from day one. Open table formats (Iceberg/Delta — see Chapter 06), standard SQL, and replaceable connectors keep exit cost low so "buy now" doesn't foreclose "build later."

Deliver it with the decision sentence from Chapter 01: "Given a four-person team and standard SaaS sources, I'd buy ingestion with Fivetran over building connectors, because time-to-value and avoiding connector maintenance dominate here. The cost is a consumption bill that grows with volume — acceptable at this scale — and I'd revisit it the moment a single high-volume source's bill exceeds what a self-hosted Kafka path would cost to run." Naming the flip condition is the whole game.

Once you've chosen what to build, the next question is whether it holds up under load and failure. Next: 08 · Scalability & Reliability.

✓ Check yourself

Answer from memory before opening each one.

1. A teammate wants to self-host Airflow to save a $60,000 yearly managed bill. Running it would take about half of one engineer's time. How do you test the proposal?

   Show answer

   Compare total cost of ownership, not the invoice. At a loaded cost of roughly $200,000 to $300,000 a year, half an engineer is $100,000 to $150,000, before the on-call burden and the features not shipped. The savings are negative unless there is a reason to own the scheduler. See the build-vs-buy framework.

2. A four-person team must load Salesforce, Stripe and HubSpot into the warehouse. Which two or three lenses carry the decision, and what is the verdict?

   Show answer

   Differentiation, total cost and time to value. Loading these sources is a commodity, building and maintaining connectors against API changes is a full-time job, and a managed connector works the same day. Buy it. See the decision scorecard.

3. A small team is told to run its own Kafka on Kubernetes for portability. What middle option gives most of the portability without most of the operations?

   Show answer

   Managed open-source Kafka, such as Confluent Cloud or a cloud provider's managed service. The protocol and client code stay portable, while the vendor carries the brokers, upgrades and capacity. You give up some control and pay a margin. See the spectrum from SaaS to in-house.

4. Per-row connector pricing is fine for 50 modest SaaS sources, but one clickstream source loads 50 billion rows a day. What do you do, and what is the test for moving it off the vendor?

   Show answer

   Keep the long tail on the managed connector and move the one expensive source to Kafka or self-hosted Connect. Move it when the annual bill for that source exceeds the loaded cost of the team needed to run the alternative, with margin. See when building is actually right.

5. A vendor is cheaper than your estimate to build, but the data is regulated and policy forbids multi-tenant SaaS. How do you decide?

   Show answer

   The compliance constraint can rule out buy whatever the cost. Ask whether the vendor offers a private deployment, customer-managed keys or in-region processing. If not, build, or buy a managed option inside your own account. See the build-vs-buy framework.

6. You buy most of the stack at the start. What choices keep a later move to build cheap?

   Show answer

   Use open table formats such as Iceberg and Parquet, standard SQL, and connectors you can replace. Then a bill or a constraint that justifies building leads to a migration project, not a rewrite. Careless buying makes lock-in the price. See the heuristic and how to defend it.

Takeaway

• Build versus buy is a decision about where finite engineering capacity goes, and it recurs at every layer of the platform.
• Compare total cost of ownership, not the invoice. Count the headcount, the on-call burden and the work not shipped.
• Buy the commodity and build the core. A layer that gives no competitive advantage is a candidate for buying.
• Managed open source sits between SaaS and self-hosting. It keeps portability and passes the operations to a vendor.
• Build wins for three reasons: scale economics, a unique requirement, and cost at extreme volume. Name which one applies.
• The default is to buy to start and build when the bill or a constraint justifies it, using open formats so the migration stays cheap.
• Buy the long tail and build the one expensive source. The stack should match the size and skill of the team that runs it.

A platform you can afford to run still has to hold up under load and failure. → Scalability & Reliability

Further reading

• Eliminating Toil — Google SRE book, 2016. A working definition of operational work that scales with load, which is the hidden cost of self-hosting.
• Fivetran usage-based pricing — Fivetran documentation. How monthly active rows drive the bill, for checking the cost curve at volume.
• BigQuery pricing — Google Cloud. On-demand and capacity compute and storage rates, a concrete input to a total-cost comparison.
• Understanding overall cost — Snowflake documentation. Storage, compute and data transfer as separate cost lines.
• Amazon S3 pricing — AWS, current. Storage and data transfer rates, including the egress line that data gravity turns into a real cost.