Requirements & Back-of-Envelope Estimation

Functional vs non-functional

Every scenario the interviewer hands you ("design the data platform for product X") is deliberately under-specified. Your first job is not to draw — it's to turn a vague prompt into a bounded problem. Split the requirements two ways.

Functional requirements are what the platform must do: which sources feed it, which consumers read from it, and which queries or products it serves. They answer "what data, for whom, to do what."

Non-functional requirements are the qualities that data must have: how fresh, how fast to query, how much of it, how correct, how available, how long retained, at what cost, under which compliance regime. In a data-systems interview these carry more design weight than the functional list, because they are what force the hard trade-offs — freshness is what pushes you toward streaming, volume is what pushes you toward a lake, correctness is what pushes you toward log-based CDC.

The clarifying questions

Ask a compact, high-leverage set — six to eight questions that each rule design space in or out. Don't interrogate; ask, listen, and let the answers steer your architecture.

• Who consumes the data, and for what? Analysts running SQL, executives reading dashboards, a model scoring in real time, another service — each implies a different serving layer.
• What are the headline queries? Point lookups by key, big scans and aggregations, or joins across many tables? This is the single biggest driver of storage choice (see access patterns).
• What's the freshness SLA — per consumer? "Real-time" almost never means the same thing across consumers. Pin it: seconds, minutes, hourly, daily. Different SLAs justify a split architecture.
• What's the volume and velocity? Events per second at average and peak, rows in the source DBs, growth rate. You'll convert this to bytes and storage in a moment.
• How correct must it be? Is an approximate dashboard fine, or is this financial reporting where a missing row is a defect? This sets the bar for delivery guarantees and reconciliation.
• How long do we retain it, and at what grain? Raw events for 30 days but aggregates forever? Retention multiplies every storage estimate.
• What's the budget and the team? A three-person team on a startup budget designs a very different platform than a 40-person org — this is where build vs buy is quietly pre-decided.
• Any compliance constraints? PII, GDPR/CCPA deletion, data residency, audit. These can force encryption, tokenization, and hard-delete paths into the design from day one.

The six properties that shape a data system

Non-functional requirements collapse into six properties. Interviewers expect you to define these crisply and — critically — to know they trade against each other. You cannot maximize all six; the requirements tell you which to favor.

Back-of-envelope estimation

Sizing is the step that turns opinions into decisions. You are not trying to be precise — you're trying to land in the right order of magnitude so you know whether this is a laptop problem or a distributed-systems problem. The chain is always the same:

1. Events/sec → bytes/day. Multiply the event rate by the average serialized size and by seconds in a day (~86,400, round to 10⁵).
2. Bytes/day → raw storage. Multiply by retention (days) to get the retained footprint.
3. Adjust for replication and compression. Multiply by the replication factor (often ×3 for a distributed store), then divide by the compression ratio (columnar formats like Parquet routinely hit 4–10× on analytic data).
4. Read QPS & peak. Estimate query concurrency, and apply a peak-to-average factor (traffic is bursty — assume peak is 2–10× average).
5. Rough cost. Storage at ~$20–25/TB/month on object storage; compute separately; watch egress (cross-region/internet transfer is often the surprise line item).

A worked sizing example

Scenario: a product emits 50 million clickstream events per day, each ~1 KB serialized as JSON. We retain raw events for 90 days and roll aggregates up indefinitely. Analysts run dashboards and ad-hoc SQL. Let's size it out loud.

Ingest rate
  50,000,000 events/day / 86,400 s  ≈  580 events/sec average
  peak (×5)                          ≈  2,900 events/sec

Raw volume per day
  50e6 events × 1 KB                 =  50 GB/day (raw JSON)

Retained raw (90 days)
  50 GB/day × 90                     =  4.5 TB raw, over the window

On object storage, columnar + compressed
  4.5 TB / 6 (Parquet compression)   ≈  0.75 TB stored (single copy)
  object storage already durable      -> no ×3 needed (managed replication)

Annualized raw (if we kept a year)
  50 GB/day × 365                    ≈  18 TB/yr raw  ->  ~3 TB as Parquet

Read side
  ~200 analyst queries/day, bursty
  concurrency peak                    ≈  10–20 simultaneous scans
  each scan touches a partition slice, not the full 0.75 TB

Rough monthly cost (storage only)
  0.75 TB × $23/TB/mo                 ≈  $17/mo  (trivial)
  => storage is NOT the cost driver here; compute + query is

Contrast the same math at scale: 50B events/day of 1 KB is 50 TB/day raw, ~8 TB/day compressed, ~700 TB retained over 90 days, and ~580,000 events/sec average — now you genuinely need partitioned streaming ingest, a lake for the cheap raw tier, and careful partitioning to avoid hot spots. Same method, wildly different platform. Always run the number before you commit to the shape.

Access patterns drive storage

Volume tells you how big; access pattern tells you what shape. The way consumers read the data is the strongest single input to the storage and serving choice you'll defend in Chapters 05 and 06.

How requirements pre-decide the trade-offs

The point of this chapter is leverage: five minutes of requirements and sizing quietly settle most of the "big" decisions before you ever debate them. Make that linkage explicit in the room — it's the clearest signal of a senior who reasons from constraints rather than from fashion.

None of this replaces the trade-off discussion — the interviewer will still pull each thread — but walking in with the requirements already mapped to leanings means you enter each fork having done the reasoning. Note that operating these pipelines reliably (backfills, late data, reconciliation) is its own discipline; when the interviewer goes there, cross-link and stay at the architecture altitude — the companion guide's reconciliation & validation chapter owns the mechanics.

Next: with requirements pinned and the platform roughly sized, we start building it — beginning at the front door, where data actually enters. Continue to 03 · Data Ingestion Pipelines.

✓ Check yourself

Answer from memory before opening each one.

1. A product emits 20 million events a day at about 2 KB each. You keep raw data for 60 days and Parquet compresses about 5 to 1. Estimate the average event rate and the stored size, and say what that tells you about the architecture.

   Show answer

   About 230 events a second on average, 40 GB a day raw, 2.4 TB over 60 days, and roughly 0.5 TB stored. That is a small dataset at a modest rate, so a batch load into a warehouse or lakehouse comfortably meets it, and streaming-first needs a freshness requirement to justify it. See back-of-envelope estimation.

2. A stakeholder says the dashboard must be real-time. Tiles load in 400 ms, but they show last night's load. Which property is being confused with which, and what do you ask?

   Show answer

   Latency is how long a query takes. Freshness is how old the data is. The dashboard has good latency and poor freshness. Ask how stale the data may be, per consumer, in seconds, minutes, hours or a day. That answer decides batch or streaming. See the six properties.

3. Why do non-functional requirements carry more design weight than the functional list in a data systems interview?

   Show answer

   They are what force trade-offs. Freshness pushes you toward streaming, volume toward a lake, correctness toward log-based CDC. Two platforms with the same functional list can look very different once those targets differ. See functional vs non-functional.

4. Analysts scan two years of history, and a model needs per-user feature lookups in under a millisecond. A product owner wants one store for both to keep things simple. What do you recommend?

   Show answer

   Split the serving layer: columnar storage for scans, and a key-value or feature store for lookups, both fed from the same curated layer. A columnar warehouse is the wrong tool for point lookups, and a row store is the wrong tool for large scans. One engine doing both is the mismatch trap. See access patterns drive storage.

5. Your sizing multiplies storage by three for replication on a self-managed cluster. A colleague applies the same factor to data in cloud object storage. Is that right?

   Show answer

   No. The factor of three models replication you operate yourself. Object storage already replicates for durability as part of the service, so counting it again overstates the footprint and the cost. Compression runs the other way and divides the size. See a worked sizing example.

Takeaway

• Requirements and a rough sizing come before any box on the board. They turn architecture from taste into argument.
• Non-functional requirements force the hard trade-offs: freshness pushes toward streaming, volume toward a lake, correctness toward log-based CDC.
• Freshness is the age of the served data and latency is the time a query takes. A warehouse can have one without the other.
• The sizing chain is events per second, bytes per day, retention, replication and compression, then peak. Round to powers of ten and say your assumptions aloud.
• At modest volume storage cost is a rounding error and compute dominates, which argues against a streaming-first build.
• The access pattern picks the store. When a scenario has both scans and point lookups, propose a split serving layer.

With requirements fixed and the platform sized, build from the front door. → Data Ingestion Pipelines

Further reading

• Service Level Objectives — Google SRE book, 2016. How to turn a vague availability or latency wish into a measurable target.
• Amazon S3 storage classes — AWS documentation. Durability and availability design targets for object storage, the basis for skipping a replication multiplier.
• Amazon S3 pricing — AWS, current. Per-GB storage, request and data transfer rates for checking the cost line of an estimate, egress included.
• Apache Parquet overview — Apache Software Foundation. The columnar layout behind the compression ratios and column pruning used in the estimates.