The front door of every data platform — how heterogeneous sources become durable, ordered, replayable data on your side of the wall, and the architecture decisions that make ingestion scale.
The source taxonomy
Ingestion is one of the two named deep-dive topics for this interview, so know it cold. Start by classifying the sources — because the source class dictates almost everything downstream: the mechanism (push or pull), the shape (files, rows, events), the delivery guarantee you can achieve, and whether you need change data capture at all.
Source class
Examples
Ingestion approach
OLTP databases
Postgres, MySQL, MongoDB behind the app
Change Data Capture — log-based or query-based. Deep dive in Ch 04.
App / clickstream events
Web & mobile analytics, user actions, telemetry
Push into a log/queue (Kafka/Kinesis/Pub-Sub) via an SDK or collector.
Third-party APIs / SaaS
Stripe, Salesforce, HubSpot, ad platforms
Pull on a schedule via a connector (Fivetran/Airbyte); rate-limited, often batch.
Files / object storage
CSV/JSON/Parquet drops, partner feeds, exports to S3/GCS
Batch file ingestion — poll a bucket/prefix, or event-triggered on object-created.
IoT / logs / devices
Sensor telemetry, application logs, edge devices
High-volume streaming push, often through an edge collector or gateway.
Classify before you design
In the room, walk the sources and tag each one: "the app DB is CDC — I'll come back to log vs query-based; clickstream is a streaming push into the log; the SaaS tools are scheduled pulls via a managed connector; the partner feed is file drops." That single pass shows structured breadth and sets up the whole ingestion layer.
Push vs pull, batch vs streaming
Two orthogonal axes govern how data crosses the boundary. Getting the vocabulary precise matters — interviewers probe it.
Push: the source (or an agent near it) sends data to you as it's produced — an SDK emitting clickstream, a device publishing telemetry, an app writing to a queue. You don't control the rate, so you must absorb bursts (backpressure, buffering). Pull: you reach out and fetch on a cadence — polling a SaaS API, listing an object-storage prefix, querying a DB for rows changed since a high-water mark. You control the rate and the batching, but you add latency (you only see data at poll time) and you can hammer the source.
Batch file ingestion moves data in bounded chunks — a nightly export, an hourly file drop, a scheduled API pull. Simple, cheap, easy to reason about and reprocess; the cost is freshness. Event streaming ingestion moves data as an unbounded flow of records through a log/queue — continuous, low-latency, replayable, but more moving parts and a harder correctness story.
Tie it back to freshness
The push/pull and batch/stream choice is downstream of the freshness SLA you pinned in Ch 02. If BI needs data hourly, a scheduled pull into a raw zone is the right, boring answer. If a fraud model needs sub-minute features, that path streams. Most real platforms are hybrid — stream the few sources that need it, batch the rest — and saying so is a senior move. The batch-vs-streaming trade-off gets its full treatment in Ch 05.
The reference architecture
Here is the ingestion layer as you'd sketch it on the whiteboard: heterogeneous sources on the left, a log/queue backbone in the middle acting as the durable buffer and ordering layer, and a landing/raw zone plus stream consumers on the right. Orchestration and a schema registry sit as cross-cutting concerns.
The key idea: decouple the sources from the consumers with a durable log in the middle. Sources produce at their own rate; consumers read at theirs; the log absorbs the mismatch, preserves order per key, and lets you replay history when a downstream job has a bug. Everything else in this chapter is a property of that backbone or of the edges around it.
The log/queue backbone
Kafka, Kinesis, Google Pub/Sub, and Redpanda all implement the same core abstraction: a partitioned, append-only, durable commit log. This is the heart of a scalable ingestion layer, and interviewers will drill the mechanics. Know these six concepts precisely.
Concept
What it is & why it matters
Partitions
Each topic splits into partitions — the unit of parallelism and ordering. More partitions = more throughput and more parallel consumers, but ordering only holds within a partition, not across them. Partition count is hard to reduce later, so size it for peak.
Keys & ordering
The record key decides the partition (hash of key → partition). Same key → same partition → guaranteed order. Choose the key to be the entity whose event order you must preserve (e.g. user_id or order_id). Order is never guaranteed globally across partitions.
Consumer groups
Consumers in a group split the partitions between them — that's how you scale reads horizontally. Each partition is consumed by exactly one member of a group; adding consumers beyond the partition count buys nothing. Different groups read the same data independently (fan-out).
Offsets
Each consumer tracks its position (offset) per partition. Committing offsets is how you record progress; where you commit relative to processing is the crux of the delivery-guarantee discussion below.
Retention
The log keeps records for a configured window (time or size) regardless of whether they've been consumed. This is what makes replay possible — reset a consumer's offset and reprocess. Set retention to cover your worst-case recovery/backfill window.
Compaction
Log compaction keeps only the latest value per key (instead of expiring by time), turning the topic into a changelog / current-state snapshot. Ideal for CDC current-state and for keyed reference data you want to bootstrap new consumers from.
The hot-partition trap
If your partition key is skewed — say most events carry the same tenant_id — one partition takes the bulk of the traffic and its consumer becomes the bottleneck while others idle. This is the ingestion face of the skew problem covered in Ch 08. Mitigate with a higher-cardinality key, or salt the key (append a bounded random suffix) when you don't need strict per-entity ordering. Interviewers love to push here — have the answer ready.
Connector frameworks & the raw zone
You rarely hand-write ingestion glue anymore; you configure a connector framework. Know the landscape and where each fits — this is also a build-vs-buy micro-decision (Ch 07).
Kafka Connect — a framework of source and sink connectors around Kafka; declarative config, scales with the cluster, hosts Debezium for CDC. The OSS default when you already run Kafka.
Airbyte — open-source (and cloud) connectors for hundreds of SaaS/DB sources; good for the long tail of third-party APIs; self-host or managed.
Fivetran — fully managed, buy-not-build; you pay per row/MAR but get connector maintenance, schema drift handling, and reliability off your plate. The modern-data-stack default for SaaS ingestion.
Custom — justified only for a source no connector covers, or extreme scale/cost where a managed per-row price is prohibitive.
Whatever the connector, land data first in an immutable raw / landing zone — object storage, partitioned by source and date, written append-only. Never transform on the way in. The raw zone is your system of record and your insurance: if a downstream transform has a bug, you replay from raw rather than re-pulling from sources that may have changed or rate-limited you. This is the "bronze" layer of the medallion pattern (Ch 06).
That config reads the Postgres write-ahead log, takes an initial snapshot then streams changes, keys each topic by primary key (so per-row order is preserved), emits deletes as tombstones, and registers schemas with the registry — the whole CDC ingest pattern in one declarative block. The strategy behind it is Chapter 04.
Schema handling & the registry
Data crossing the boundary has a schema, and schemas drift — a source adds a field, renames one, changes a type. How you handle this splits into two philosophies.
Schema-on-write validates and enforces structure at ingest: the record must conform before it lands. You catch bad data early and downstream consumers get guarantees, but a rigid schema rejects data or breaks when the source changes. Schema-on-read lands whatever arrives (often as JSON) and imposes structure at query time. Flexible and resilient to drift, but you push the pain — and the surprises — to consumers. Lakes lean schema-on-read for the raw zone; warehouses enforce schema-on-write for curated tables. A common, pragmatic split: schema-on-read into raw, schema-on-write into curated.
A schema registry (Confluent Schema Registry, AWS Glue Schema Registry) is the mechanism that makes streaming ingest safe. Producers register an Avro / Protobuf / JSON-Schema definition; the registry assigns a version and enforces a compatibility mode (backward, forward, full) so a producer can't ship a breaking change that would blow up consumers. Records carry a schema ID, not the full schema, keeping them compact.
Compatibility modes in one line
Backward — new consumers can read old data (safe to add optional fields, drop fields). Forward — old consumers can read new data. Full — both. Pick backward as the default for analytics; it lets producers evolve while historical replay still works. Schema evolution gets fuller treatment in Ch 09, and enforcing producer/consumer contracts in CI is covered in the companion guide's validation chapter.
Delivery guarantees
This is where ingestion interviews get real. Three levels of guarantee, and you must know why the middle one plus a design trick beats chasing the third directly.
Guarantee
Meaning
Cost
At-most-once
Every record delivered zero or one times — may drop on failure.
Cheapest, but loses data. Rarely acceptable for a platform of record.
At-least-once
Every record delivered one or more times — never lost, but may duplicate on retry.
The practical default. Duplicates are the price of never losing data.
Exactly-once
Every record's effect applied precisely once.
Expensive and often impossible end-to-end across heterogeneous systems.
The senior answer: effectively-once
Don't promise true end-to-end exactly-once — instead combine at-least-once delivery with an idempotent sink. The producer/consumer never drops (at-least-once), and the sink deduplicates on a stable key so re-delivering a record is a no-op. Net effect: exactly-once outcomes without the fragility. Concretely: attach a deterministic dedup_key (or use the source primary key + LSN/offset), and land via an idempotent upsert/MERGE or insert-if-not-exists. "At-least-once plus idempotent sink equals effectively-once" is the sentence that scores. Note that the sink itself must be idempotent — commit offsets only after the write succeeds, or a crash between write and commit reprocesses cleanly.
Ordering matters alongside delivery: if you keyed the topic by entity, per-entity order holds within a partition, so an idempotent sink applying updates in offset order converges to the correct current state even if some are redelivered. The merge mechanics that make this land correctly in a warehouse live in the companion guide's incremental processing chapter — reference it rather than re-teaching the SQL here.
Backpressure, batching & dead letters
At scale, the failure modes are about flow and bad data, not happy-path throughput. Three mechanisms you should name.
Backpressure. When a consumer can't keep up, the system must slow the producer or buffer — not silently drop. The durable log is itself the buffer: producers keep writing, retention gives consumers room to catch up, and lag (the offset gap) is your key health metric. If lag grows unbounded, you scale consumers (up to the partition count) or add partitions.
Batching & partitioning for throughput. Producers batch records before sending (larger, fewer requests) and compress the batch; this is the single biggest throughput lever. Partitioning spreads load across brokers and consumers. Tune batch size and linger against your latency budget — bigger batches mean higher throughput but more latency.
Dead-letter queue (DLQ). Records that can't be decoded, fail schema validation, or repeatedly error must not block the pipeline. Route them to a DLQ (a side topic or table) for inspection and replay, and alert on its depth. Without a DLQ, one poison record halts a partition — a classic outage.
Here's a producer configured for effectively-once, ordered, batched ingest — the config choices matter as much as the code.
producer.py
from confluent_kafka import Producer
producer = Producer({
"bootstrap.servers": "broker:9092",
"enable.idempotence": True, # no duplicate writes on retry
"acks": "all", # wait for all in-sync replicas -> durable
"max.in.flight.requests.per.connection": 5, # safe w/ idempotence, keeps order
"linger.ms": 20, # batch window: trade latency for throughput
"batch.size": 262144, # 256 KB batches
"compression.type": "zstd", # shrink bytes on the wire
})
def emit(event: dict) -> None:
# key by entity -> same partition -> per-entity ordering preserved
producer.produce(
topic="clickstream.events",
key=event["user_id"],
value=serialize(event), # Avro/Protobuf via schema registry
)
# flush on shutdown so nothing is lost in the buffer
producer.flush()
Ingestion vs orchestration
Ingestion gets data onto the platform durably and in order. Scheduling the batch pulls, wiring dependencies between jobs, and retrying failed runs is orchestration — a different concern owned by the companion guide's orchestration chapter. In the interview, land data reliably here and cross-link there; don't blur the two.
The seniority ladder
How an ingestion answer reads at each level — a quick self-check for the depth an interviewer expects.
Level
An ingestion answer sounds like…
L1 · Foundational
"Pull the data with a connector and load it into the warehouse on a schedule." Correct for simple cases; no mention of ordering, guarantees, or failure.
L2 · Practitioner
Reaches for Kafka or Fivetran appropriately, lands in a raw zone, knows at-least-once vs exactly-once, mentions a DLQ.
L3 · Senior
Classifies sources, picks push/pull and batch/stream per source against the freshness SLA, keys topics for ordering, achieves effectively-once with an idempotent sink, handles schema drift with a registry.
L4 · Staff
Reasons about hot partitions and key design, sizes partition count for peak, sets retention to the recovery window, designs replay/backfill, weighs connector build-vs-buy on TCO, and draws the boundary to CDC, orchestration, and reconciliation cleanly.
The single richest source — an OLTP database you must replicate continuously without hammering it — has its own strategy space. That's the next named deep dive: 04 · Change Data Capture Strategies.
✓ Check yourself
Answer from memory before opening each one.
A topic is keyed by tenant ID and one tenant produces 80 percent of the events. What do you see, why do extra consumers not help, and what are two mitigations with their costs?
Show answer
One partition carries most of the traffic and its consumer falls behind while the others idle. A partition is read by one member of a group, so more consumers cannot split it. Use a higher-cardinality key, or salt the key with a bounded random suffix. Both give up strict per-tenant ordering to some degree. See the log backbone.
A consumer group has 12 members on a topic with 8 partitions, and lag keeps growing. A teammate wants to add 4 more consumers. What happens?
Show answer
Nothing improves, because each partition is read by one member of the group and 4 of the 12 are already idle. The fix is faster consumers or more partitions, and partition count is hard to reduce later, so size it for peak. See the log backbone.
A consumer commits its offset, then writes the record to the sink, and the process crashes between the two steps. What delivery guarantee did you build, and how do you fix it?
Show answer
At-most-once: the offset moved past a record the sink never received, so it is lost. Commit after the write succeeds. That gives at-least-once, so a crash can replay a record, and the sink must absorb the duplicate. See delivery guarantees.
Why do we aim for at-least-once delivery with an idempotent sink instead of promising end-to-end exactly-once, and what must the sink have for this to work?
Show answer
True exactly-once across heterogeneous systems is expensive and often impossible. At-least-once never loses a record, and an idempotent sink makes a redelivery a no-op. The sink needs a stable dedup key, such as the primary key with the offset, and an upsert or merge. With per-key ordering, applying records in offset order converges to the right state. See delivery guarantees.
A partner feed adds and renames fields about once a month. Where do you validate it, and which schema approach do you use at each stage?
Show answer
Land it as received in the raw zone with schema-on-read so drift never blocks ingestion. Enforce structure with schema-on-write when you build curated tables, where a mismatch is caught before consumers see it. For streams, a registry with backward compatibility stops a breaking change at the producer. See schema handling.
At 3 a.m. one undecodable record halts a partition and lag climbs. What was missing, and what do you build?
Show answer
A dead-letter path. Route records that fail decoding or validation to a side topic or table, keep the pipeline moving, alert on its depth, and replay after a fix. Because the raw zone is immutable, you can also rebuild downstream state from it. See backpressure, batching and dead letters.
Takeaway
Classify each source first. The class picks the mechanism (CDC, push into a log, scheduled pull, file poll) and the freshness SLA picks batch or stream per source.
A durable log between sources and consumers absorbs rate mismatch, keeps per-key order, and lets you replay after a bug.
Order holds only within a partition. Key by the entity whose order matters, and know that a skewed key makes a hot partition that extra consumers cannot fix.
Land data unchanged in an immutable raw zone, so a bad transform is fixed by replaying rather than re-pulling from sources.
A schema registry with backward compatibility lets producers evolve without breaking consumers. Use schema-on-read for raw and schema-on-write for curated.
Promise at-least-once delivery with an idempotent sink and offsets committed after the write, not end-to-end exactly-once.
Consumer lag is the health metric, batching is the main throughput lever, and a dead-letter path keeps one bad record from halting a partition.
The richest source is a live database you must copy without hammering it, and that has its own strategy space. → Change Data Capture
Further reading
Apache Kafka documentation: Design — Apache Software Foundation. Partitions, producers, consumers, delivery semantics and compaction as the project defines them.
Kafka log compaction — Apache Kafka documentation. How a topic keeps the latest value per key, the basis for bootstrapping consumers from current state.
Debezium connector for PostgreSQL — Debezium project documentation. Snapshot modes, logical decoding and the topic and key conventions used in the example config.