Scalability & Reliability

Scale and reliability as design axes

By this point in a session you have an architecture on the board and the big trade-offs decided. Now the interviewer stress-tests it: "what happens at 10× volume?", "what if this node dies mid-load?", "one customer is 40% of your traffic — what breaks?" These questions live at every layer, so treat scalability and reliability as cross-cutting concerns you can reason about anywhere on the diagram, not a bolt-on section.

The two axes trade against each other and against cost. More replicas buy durability and availability but cost money and can hurt consistency. More partitions buy throughput but create small-files and skew problems. The senior signal is naming which property the requirements actually demand — a BI warehouse tolerates hours of downtime and eventual consistency; a fraud-feature path does not — and spending only where it's required.

Partitioning & sharding

Partitioning is the foundation of horizontal scale: split data into independent chunks so many nodes work in parallel and each query touches only the relevant slice. It shows up everywhere — Kafka topic partitions, warehouse micro-partitions and clustering, object-storage layouts (s3://.../dt=2026-09-12/), and OLTP shards. The two dominant strategies:

• By key (hash). Hash a key (user_id, account_id) to assign a partition. Even distribution and O(1) point lookups by key — but range scans hit every partition, and a single hot key can't be split further.
• By time (range). Partition by event date/hour. Ideal for analytics — most queries filter by time, so the engine prunes to a few partitions, and old data ages out cheaply. But the current time-partition is a write hotspot, and time queries can skew.

Most real platforms combine them: partition by time for pruning and retention, cluster/sub-partition by a high-cardinality key within each partition for lookup performance. In Kafka, the partition key also controls ordering — records with the same key land on the same partition in order, which is why you key CDC streams by primary key (see Chapter 04).

The hot-partition & skew problem

Partitioning assumes even distribution. Reality rarely obliges: one enterprise customer is 40% of your events, NULL collapses into one partition, or a celebrity's user_id gets a spike. That partition becomes a hotspot — it lags, its consumer falls behind, its reducer runs long after all others finish, and the whole job's latency is set by the slowest partition. Data skew is the single most common scaling failure interviewers probe, so have the mitigations ready.

  EVEN distribution (healthy)            SKEWED distribution (hot partition)
  ┌─────┬─────┬─────┬─────┐              ┌─────┬─────┬─────┬───────────────┐
  │ P0  │ P1  │ P2  │ P3  │              │ P0  │ P1  │ P2  │      P3       │
  │ ▓▓▓ │ ▓▓▓ │ ▓▓▓ │ ▓▓▓ │              │ ▓   │ ▓   │ ▓   │ ▓▓▓▓▓▓▓▓▓▓▓▓▓ │
  └─────┴─────┴─────┴─────┘              └─────┴─────┴─────┴───────────────┘
   all consumers keep pace                P3 = "whale" tenant; its consumer
   job latency = avg partition            lags, job latency = P3, others idle

  MITIGATION — salting the key:  key = tenant_id            (all whale → P3)
                                 salted = tenant_id + "#" + (hash(row) % N)
                                 → whale now spread across P3a..P3n
                                 (aggregations must re-combine the N buckets)

Mitigations, roughly in order of preference:

• Better key design. Choose a higher-cardinality or composite key so no single value dominates. Adding a natural sub-dimension (region, event_type) often flattens the distribution for free.
• Salting. Append a small random/hashed suffix to the hot key to fan it across N partitions, then re-aggregate the N buckets at read time. Standard for skewed joins in Spark and for hot Kafka keys where strict per-key ordering isn't required.
• Isolate the whale. Route the one giant tenant to a dedicated partition, topic, or even pipeline so it can't starve everyone else — a common multi-tenant pattern.
• Engine-level handling. Spark's adaptive query execution (AQE) splits skewed join partitions automatically; use it before hand-rolling salting.
• Two-phase aggregation. Pre-aggregate locally (map-side / partial) so the skewed key's volume shrinks before the final shuffle.

Horizontal, vertical & elastic scaling

Vertical (scale up) — a bigger machine. Simple, no distribution complexity, but a hard ceiling and a single failure domain. Fine for the metadata DB or a modest warehouse. Horizontal (scale out) — more machines. Effectively unbounded and fault-tolerant, but requires partitioning, coordination, and shuffle. Every large data system is horizontal; the question is where you accept the coordination cost.

The modern enabler is storage/compute separation: data lives in object storage (S3/GCS) and stateless compute scales independently against it. This is why Snowflake, BigQuery, Databricks, and Trino can spin compute up for a heavy query and down to zero after — you pay for storage always and compute only when running. It decouples the two scaling axes so a storage-heavy, compute-light workload doesn't overpay, and it's the architectural reason autoscaling is cheap on these platforms. (More in Chapter 06.)

Autoscaling adjusts compute to load — Snowflake multi-cluster warehouses spinning up for concurrency, Kubernetes-based Spark/Flink scaling executors, serverless query engines. The caveats worth naming: cold-start latency, the cost of scaling reactively too late (queues build), and runaway spend if you don't cap it. Autoscaling is a cost-shaping tool, not a substitute for fixing a skewed or under-partitioned job — it'll happily scale a broken query's bill to the moon.

Reliability: durability, fault tolerance, backpressure

Reliability is a bundle of distinct guarantees. Name them separately; conflating "reliable" hides which one the requirements actually need.

• Durability — data, once accepted, is not lost. Achieved by replication: object stores keep multiple copies (S3's eleven-nines durability), Kafka replicates each partition to N brokers with a configurable min.insync.replicas and acks, and warehouses replicate under the hood. Durability is cheap and near-absolute on managed cloud storage — the interesting failures are upstream, before data is durably landed.
• Availability — the system answers requests now. Set a target against the requirement: BI can tolerate 99% (hours/month down); an operational serving layer feeding a product needs 99.9%+. Availability comes from redundancy and no single point of failure — replicate the metadata store, run multiple query nodes, avoid the one Airflow scheduler being the whole platform's SPOF.
• Fault tolerance & graceful degradation — a component failing degrades service rather than collapsing it. Retries with backoff and jitter, dead-letter queues for poison records (Chapter 03), circuit breakers, and serving slightly stale data rather than erroring. A good platform under partial failure keeps the freshness clock ticking on healthy paths and isolates the broken one.
• Backpressure & flow control — when a downstream can't keep up, the system slows the source instead of dropping data or exhausting memory. A durable log (Kafka/Kinesis) is the classic buffer: producers write at their rate, consumers read at theirs, and the log absorbs the mismatch up to its retention. Flink and reactive frameworks propagate backpressure explicitly; a naive HTTP-push ingestion path without a buffer has no backpressure and sheds load under a spike.
• Idempotency & exactly-once — reprocessing or retrying doesn't double-count. The building block for tolerating the retries that fault tolerance depends on: idempotent writes keyed on a dedup/business key, or an at-least-once source plus an idempotent sink to get effectively-once.

Consistency for data platforms

Distributed systems can't have it all, and data platforms sit on the relaxed end of the spectrum. Two framings worth deploying:

• CAP. Under a network partition you choose availability or consistency. Analytics platforms almost always choose availability — a dashboard returning slightly stale numbers beats one returning an error.
• PACELC. The sharper framing: even when there's no partition (Else), you trade latency vs consistency. This maps directly onto data platforms — you accept a source-to-analytics lag (latency of consistency) to get cheap, high-throughput, available reads. Naming PACELC over plain CAP is a seniority tell because it captures the everyday trade, not just the failure case.

The practical consistency question for a platform is source-to-analytics lag: how stale is the warehouse relative to the source of truth? A batch pipeline is hours behind by design; CDC-plus-streaming is seconds-to-minutes behind. That lag is your consistency model — the platform is eventually consistent with the OLTP source, converging as data flows through ingestion, transform, and serving. Set the freshness SLA (Chapter 02) to the lag each consumer tolerates, and don't pay streaming's cost for a consumer that's happy hours behind.

Read-after-write matters at boundaries: after a MERGE completes, do queries see the new state atomically? Open table formats (Iceberg/Delta) give ACID snapshot isolation on the lake so readers never see a half-written commit — a concrete reason the lakehouse formats exist. Strong consistency within the warehouse, eventual consistency between source and warehouse: that's the normal shape, and saying it precisely shows you know where each guarantee lives.

Multi-region & disaster recovery

DR is measured by two numbers — state them before proposing a design, because they set the cost:

• RPO (Recovery Point Objective) — how much data you can afford to lose, measured in time. RPO of one hour means hourly backups/replication are enough; RPO near zero demands continuous cross-region replication.
• RTO (Recovery Time Objective) — how fast you must be back. RTO of a day tolerates a restore-from-backup; RTO of minutes demands a warm/hot standby.

A crucial data-platform insight: much analytics data is reproducible — it's derived from a source system by pipelines. If you can replay from the raw/bronze layer or re-ingest from source, your effective RPO for the derived marts is bounded by how fast you can reprocess, not by whether you replicated the gold tables. That's why cheap backup-and-restore is often the right DR posture for a warehouse, and why the immutable raw landing zone earns its keep. Backfilling at scale is itself a reliability capability — design ingestion (Chapter 03) so you can replay a day or a region without hammering the source or double-counting, which is again where idempotency pays off.

Bottleneck analysis in a design

The evolution phase always ends here: "walk me through 10× — what breaks first?" The strong answer isn't "we'd scale everything." It's a systematic sweep that names the first constraint, because a system's throughput is set by its narrowest stage.

1. Follow the data left to right and ask where volume or QPS hits a limit first: source extraction rate → ingestion/log throughput → transform compute → warehouse concurrency → BI query load.
2. Find the stage that scales worst. Stateless, storage/compute-separated stages scale easily; the bottleneck is usually the stateful or single-instance piece — the source database's CDC read capacity, a single-partition hot key, the orchestrator scheduler, a metadata store, or warehouse slot/credit limits under concurrency.
3. Name the mitigation for that specific stage — more Kafka partitions and consumers, split the hot key, separate compute pools per workload, add read replicas, autoscale the warehouse — and then re-ask what the next bottleneck becomes, since fixing one moves it downstream.

With scale and reliability reasoned through, the last cross-cutting concern is making the platform trustworthy and usable. Next: 09 · Modeling, Schema Evolution & Governance.

✓ Check yourself

Answer from memory before opening each one.

1. One tenant produces 40 percent of the events in a CDC stream keyed by primary key, and its partition is falling behind. A colleague suggests salting the key. Does that work here?

   Show answer

   No. Salting spreads one key across many partitions, which breaks per-key ordering, and the merge at the sink relies on that order to converge to the right state. Use a better composite key where possible, or isolate the large tenant on its own topic or pipeline. See the hot-partition and skew problem.

2. A Spark join runs for four hours because 199 of 200 tasks finish in minutes. What is the cause, and what do you try before hand-rolling a fix?

   Show answer

   One join key dominates, so a single task shuffles most of the data and sets the job's latency. Turn on adaptive query execution, which splits skewed join partitions, before writing salting logic. If that is not enough, pre-aggregate partially or salt the key. See the hot-partition and skew problem.

3. A clickstream table is queried mostly by date range and sometimes by user. How do you partition it, and what goes wrong if you partition by hour, country and device?

   Show answer

   Partition by time so queries prune to a few partitions and old data ages out cheaply, then cluster or sort by user within each partition. Partitioning by hour, country and device yields millions of tiny files, which slows reads and bloats metadata. Compaction repairs it. See partitioning and sharding.

4. A skewed query is slow, and someone turns on autoscaling for the warehouse. The query is no faster and the bill doubles. Why?

   Show answer

   Autoscaling adds compute, but the work is stuck on one overloaded partition, so extra nodes sit idle. It shapes cost against load. It does not fix a skewed or under-partitioned job. Fix the key or the query first. See horizontal, vertical and elastic scaling.

5. Derived marts can be rebuilt from an immutable raw layer. The business sets an RPO of one hour for them. What DR posture do you propose, and why is it cheaper than it sounds?

   Show answer

   Backup and restore, or replay from raw. The effective loss window for derived tables is bounded by how fast you can reprocess, not by whether you replicated them. Keep the raw layer durable and replayable, and spend on replication only for sources that cannot be re-ingested. See multi-region and disaster recovery.

6. The interviewer says: at 10 times volume, what breaks first? Describe how you find the answer, and give one example that is not "everything needs more nodes".

   Show answer

   Follow the data from source extraction through ingestion, transform, warehouse and BI, and look for the stateful or single-instance stage that scales worst. For example, the CDC read from the source database through one replication slot, which competes with production traffic. Name the mitigation, then ask what the next bottleneck is. See bottleneck analysis.

Takeaway

• Scale and reliability are sets of named mechanisms, each aimed at a specific failure. Name the mechanism and the failure it solves.
• Partition by key for even spread and lookups, by time for pruning and retention, and combine them. Over-partitioning creates small files.
• Skew sets a job's latency by its slowest partition. Try key design, adaptive execution and isolation before salting, and remember salting breaks ordering.
• Storage and compute separation makes scaling cheap, but autoscaling cannot fix a skewed job. It only shapes cost.
• Reliability is several guarantees: durability, availability, graceful degradation, backpressure and idempotency. Say which one the requirement needs.
• A platform is eventually consistent with its source. Set each freshness SLA to the lag the consumer tolerates, and let table formats give atomic reads inside the lake.
• Set RPO and RTO first. Derived data that can be replayed from raw usually needs only backup and restore.

A platform that scales and survives still has to stay trustworthy as schemas change and people depend on it. → Modeling, Schema Evolution & Governance

Further reading

• Addressing Cascading Failures — Google SRE book, 2016. Load shedding, retries with backoff and graceful degradation under overload.
• Handling Overload — Google SRE book, 2016. Per-client limits, load shedding and why a system must degrade rather than collapse.
• Spark SQL performance tuning — Apache Spark documentation. Adaptive query execution and its skew-join handling.
• Kafka replication — Apache Kafka documentation. In-sync replicas, acks and the durability trade-offs behind min.insync.replicas.
• Consistency Tradeoffs in Modern Distributed Database System Design — Abadi, IEEE Computer, 2012. The paper that introduced PACELC.