Section D · Cross-Cutting Concerns

Modeling, Schema Evolution & Governance

What turns a pile of tables into a platform people trust: how you model it, how it changes safely over time, and how you keep it discoverable, compliant, and cost-controlled at scale.

Modeling on the platform

Once storage and processing are settled, the interviewer often asks "how would you model this?" — and the answer is never one shape for the whole platform. Different layers want different models. The medallion layout from Chapter 06 is the frame: raw/bronze is source-shaped and append-only, silver is cleaned and conformed, and gold is modeled for consumption. The modeling question is really "what shape does gold take?", and there are three defensible answers.

Approach	What it is	Strengths	Costs	Fits when
Dimensional (star / snowflake)	Facts (events/measures) + conformed dimensions (entities); snowflake normalizes dimensions further.	Intuitive for BI, fast aggregations, well-understood grain, great with a semantic layer.	ETL to conform dimensions; slowly-changing-dimension (SCD) logic; less flexible to source change.	BI and analytics is the primary consumer; stable business entities; you want a semantic model on top.
Data Vault (hubs, links, satellites)	Hubs = business keys, links = relationships, satellites = descriptive/temporal attributes.	Auditable, insert-only, absorbs source change and new sources gracefully; parallel loads.	Verbose (many joins), needs a presentation layer (marts) on top before BI can use it; steeper learning curve.	Many volatile sources, heavy compliance/audit needs, large teams integrating over time.
One Big Table / wide denormalized	A single wide table (or a few) with everything pre-joined, often per event or per entity-day.	No joins at query time; blazing on columnar engines; simplest for consumers and for ML features.	Redundancy, expensive to update, dimension changes force rewrites, can hide grain bugs.	Read-heavy columnar warehouses/lakehouses, ML feature tables, dashboards over a fixed grain.

The senior framing

These aren't rivals across the platform — they're layers. A common staff-level answer: Data Vault (or raw+silver) as the integration layer for auditability and source resilience, dimensional marts for BI, and wide OBT tables materialized for the heaviest dashboards and ML features. Columnar storage and cheap compute have made denormalization far more attractive than the classic Kimball era assumed — pre-joining is often cheaper than joining at query time when reads dominate. Say why the layer picks the model, and you've answered the question at the right altitude.

Modeling for BI vs ML features

The two big consumers of gold pull the model in different directions, and naming the split shows you've served both.

BI/analytics wants conformed dimensions, a stable grain, and a semantic/metric layer (dbt's semantic layer, LookML, Cube) so "revenue" means one thing everywhere. Correctness and consistency beat latency; SCD Type 2 preserves history for as-of reporting.
ML features want wide, per-entity feature vectors with strict point-in-time correctness — a feature must reflect only what was known at the label's timestamp, or you leak the future into training. This is why a feature store (Feast, Tecton, Databricks/Vertex feature stores) exists: it enforces one definition for offline training and online serving, preventing train/serve skew.

Say the magic words

For any ML-features question, name point-in-time correctness and train/serve skew. "I'd compute features from the same transformation for both the offline training set (as-of the label time) and the online serving path — a feature store gives me that single definition and low-latency lookups" is the answer that lands. The join-to-labels-without-leakage problem is the ML analogue of the correctness stories in ingestion and CDC.

Schema evolution: change without breaking consumers

Schemas change constantly — a new column, a renamed field, a tightened type. The design question is: how does a producer change a schema without breaking every downstream consumer? The dividing line is additive vs breaking.

Additive (safe): adding an optional/nullable field, adding an enum value in some encodings, widening a type. Old readers ignore what they don't know; old data still parses.
Breaking (dangerous): removing or renaming a field, changing a type incompatibly, making an optional field required, changing semantics of an existing field (the silent killer — same name, new meaning).

A schema registry (Confluent Schema Registry, AWS Glue Schema Registry, or the schemas embedded in Avro/Protobuf) enforces a compatibility mode at write time, so a bad change is rejected before it corrupts the pipeline rather than discovered in a broken dashboard.

Compatibility mode	Guarantees	Allowed change	Upgrade order
Backward	New schema reads old data	Delete fields, add optional fields	Upgrade consumers first
Forward	Old schema reads new data	Add fields, delete optional fields	Upgrade producers first
Full	Both directions	Add/remove optional fields only	Any order
None	No checks	Anything	Coordinate manually (avoid)

Backward is the sensible default for analytics: you can redeploy consumers to understand the new schema, then let producers emit it. On the storage side, open table formats make evolution a first-class, metadata-only operation. Iceberg tracks columns by a stable field ID (not by name or position), so a rename is a metadata change and never rewrites data files or breaks old readers; it also supports partition evolution — changing how a table is partitioned without rewriting history. Plain Parquet evolves too (add columns, read old files as null) but without the transactional guarantees, which is exactly the "swamp" risk a table format removes.

iceberg-evolve.sql

-- Additive + rename, both metadata-only in Iceberg (no file rewrite):
ALTER TABLE orders ADD COLUMN loyalty_tier STRING;      -- old readers see NULL
ALTER TABLE orders RENAME COLUMN amt TO amount_usd;     -- tracked by field ID

-- Partition evolution: start partitioning new data by day without touching history
ALTER TABLE events ADD PARTITION FIELD days(event_ts);

-- Time-travel to read the table as of a prior snapshot (audit / reproducibility)
SELECT * FROM orders FOR SYSTEM_VERSION AS OF 8123456789012345678;

The change that passes the registry and still breaks you

Compatibility modes catch structural breaks, not semantic ones. Repurposing status = 'active' to mean something new, or switching a timestamp from local to UTC, keeps the same schema and sails through the registry while silently corrupting every downstream metric. Semantic change is a breaking change even when the shape is identical — which is exactly the gap data contracts close.

Data contracts

A data contract is an explicit agreement between a data producer and its consumers covering three things: schema (fields and types), semantics (what each field means, units, allowed values, nullability, the grain), and SLA (freshness, availability, and how change is communicated). It moves data quality upstream — the producer owns the shape and meaning of what they emit, instead of a fleet of downstream teams reverse-engineering it and building fragile defensive logic.

The contract is worth little unless it's enforced. Two enforcement points:

In CI — a schema/semantic change to a source is checked against the contract in the producer's pull request, so a breaking change is blocked before merge (registry compatibility check + expectation tests on sample data).
At ingest / at the gate — incoming data is validated against the contract on the way into the platform; violations route to a dead-letter path rather than poisoning the warehouse.

data-contract.yaml

contract: orders.v2
owner: checkout-team
sla:
  freshness: 15m          # rows land within 15 minutes of the source commit
  availability: 99.9%
schema:
  - name: order_id
    type: string
    unique: true
    nullable: false
  - name: amount_usd       # renamed from amt; units are USD, not cents
    type: decimal(12,2)
    nullable: false
    constraint: ">= 0"
  - name: status
    type: string
    allowed: [pending, paid, shipped, cancelled]   # semantics pinned
  - name: event_ts
    type: timestamp
    timezone: UTC          # semantic guarantee, not just a type
compatibility: backward     # enforced in producer CI

Cross-link, don't re-teach

The mechanics of turning these expectations into runnable tests — assertions, thresholds, reconciliation against the source, quarantine of bad rows — belong to the companion guide. Point the interviewer there and stay at the architecture altitude: see Reconciliation & Validation in Production Data Pipelines for the test framework, and cite the contract as where the expectations come from.

Catalog, lineage & discovery

At platform scale the practical failure isn't that data is missing — it's that nobody can find it, trust it, or tell what breaks if they change it. A data catalog (DataHub, OpenMetadata, Amundsen, Unity Catalog, Collibra, or the cloud-native Glue/Dataplex catalogs) is the metadata backbone that fixes this:

Discovery — search datasets by name, tag, domain, owner; see descriptions, freshness, and popularity so consumers pick the blessed gold table, not a stale copy.
Lineage — column- and table-level graph of what feeds what, harvested from query logs, dbt/orchestrator metadata, and connectors. Lineage answers "if I change this column, what dashboards break?" and "where did this wrong number come from?" — the two questions that dominate real incidents.
Ownership & classification — every dataset has an owner and tags (domain, PII level, tier), which is what makes the access and compliance controls below enforceable.

Lineage is also the substrate for observability at scale: freshness, volume, and schema monitors (Monte Carlo, dbt tests, Great Expectations) attach to catalog assets so an anomaly is triaged with its downstream blast radius already known.

Quality, access control & PII

Three governance concerns cluster here because they're all about making a shared platform safe rather than just functional.

Quality/observability at platform scale. Beyond per-pipeline tests, the platform needs standing monitors — freshness SLOs, row-volume anomaly detection, null-rate and distribution drift, and schema-change alerts — reported per dataset and tied to the catalog and contracts. The goal is to detect the bad batch before a consumer does. (Delivery-semantics and pipeline-level reliability live in Chapter 08 and the pipelines guide's reliability chapter.)

Access control. Prefer role- or attribute-based access (RBAC/ABAC) over per-user grants, ideally driven by classification tags rather than table-by-table ACLs. Column- and row-level security (and dynamic masking) let one physical table serve many audiences — analysts see masked emails, the fraud team sees more. Unity Catalog, Snowflake, and BigQuery all express this natively; the design point is governance by policy on tags, so a newly-tagged PII column is protected automatically.

PII, security & compliance. GDPR and CCPA make personal data a first-class design constraint, not an afterthought:

Masking — redact or partially obscure at read time (dynamic data masking) so most consumers never see raw PII.
Tokenization / pseudonymization — replace PII with tokens, keeping the sensitive values in a separate vault; joins still work on the token, but a lake breach leaks nothing usable.
Retention & deletion — enforce time-based retention and support the right to erasure. This is genuinely hard on immutable, append-only lakes: you need row-level delete support (Iceberg/Delta merge-on-read deletes, or crypto-shredding — destroy the per-subject encryption key so the data is unrecoverable) plus a way to prove propagation across derived tables using lineage.

Interview trap: "how do you delete a user from a data lake?"

"Just DELETE" ignores that raw Parquet is immutable and that the user's data has fanned out into a dozen derived tables. The credible answer: table formats with row-level deletes (or partition-scoped rewrites) for the tables that must physically drop rows, crypto-shredding for cold/immutable archives, and lineage-driven propagation so you can enumerate and prove every place a subject's data landed. Tie it back to retention policy so deletion is routine, not a fire drill.

Cost governance

On elastic cloud platforms, cost is a design property that erodes silently — an unpartitioned table that full-scans on every query, a dashboard auto-refreshing every minute, a dev running SELECT * on a petabyte. Seniority shows in making cost visible and owned:

Chargeback / showback — attribute spend to teams via tags, warehouses, or query labels so the team generating the cost sees the bill. Visibility alone kills most waste.
Right-sizing — separate warehouses/clusters per workload, auto-suspend idle compute, cap query bytes/credits, and set retention on cold data (or tier it to cheaper storage) instead of keeping everything hot forever.
Design-time levers — partitioning and clustering to prune scans, materializing heavy aggregations once instead of recomputing per query, and choosing batch over streaming when the freshness SLA allows (Chapter 05) are the biggest cost decisions, made long before the invoice arrives.

Governance is the platform's usability layer

Modeling, schema evolution, contracts, catalog, security, and cost aren't separate checkboxes — they're the difference between a platform people trust and reach for, and a data swamp with a big bill. When an interviewer probes governance, they're checking whether you've operated a platform at scale, not just built one. The tell is whether you connect these back to concrete failure modes: the broken dashboard, the leaked PII, the surprise bill.

The seniority ladder

Where an answer on modeling and governance lands:

Level	Sounds like
L1 · Foundational	Builds a star schema; knows PII should be protected but treats governance as someone else's job.
L2 · Practitioner	Chooses a model per layer, uses a schema registry with a compatibility mode, adds column masking and a catalog.
L3 · Senior	Enforces data contracts in CI and at ingest, designs point-in-time-correct ML features, and has a real erasure/retention story on an immutable lake.
L4 · Staff	Treats modeling, contracts, lineage, policy-on-tags security, and chargeback as one governance system tied to failure modes and cost — and knows which controls to skip early to avoid over-engineering a small platform.

Next, everything in Sections A–D comes together on a whiteboard: 10 · The Design Session walks the six-step method end-to-end on a full platform scenario.

✓ Check yourself

Answer from memory before opening each one.

A producer starts using status = 'active' to include trial accounts. The schema is unchanged and the registry accepts it. What breaks, and which control is meant to catch it?

Show answer
Every downstream metric that counts active accounts changes meaning silently. Compatibility modes check structure, not semantics. A data contract that pins allowed values and meaning, enforced in the producer's CI, is the control that catches it. See schema evolution and data contracts.
Why is renaming a column in an Iceberg table a metadata change, while the same rename in a pile of plain Parquet files is risky?

Show answer
Iceberg tracks columns by a stable field ID, not by name or position, so a rename edits metadata and leaves data files and old readers alone. Plain files match by name, so readers on the old name see nulls, and nothing coordinates the change. See schema evolution.
Under backward compatibility, a producer wants to add a new required field with no default. What happens, and what do you ask for instead?

Show answer
The registry rejects it, since a consumer on the new schema could not read old records that lack the field. Ask for an optional field or one with a default. Under backward mode you upgrade consumers first, then producers. See schema evolution.
An ML model scores well offline and badly in production. The training set joined each label to the latest feature values. What is wrong, and how do you fix the pipeline?

Show answer
The join leaked the future: features reflected what was known after the label time, and the serving path computes them differently, which is train and serve skew. Build the training set with point-in-time-correct joins, and use one feature definition for offline and online, as a feature store provides. See modeling for BI vs ML.
A regulator asks you to delete one user from an append-only lake whose data has fanned out into a dozen derived tables. How do you do it, and how do you prove it?

Show answer
Use a table format with row-level deletes or scoped rewrites for tables that must drop rows, and crypto-shredding for cold archives by destroying the subject's key. Use lineage to list every table the user's data reached, and verify each. Retention policy makes this routine. See quality, access control and PII.
Your company has many volatile sources, audit obligations and BI dashboards, plus an ML team. Which model do you choose for each layer?

Show answer
Use layers, not one model. Data Vault or conformed silver as the integration layer for audit and source change, dimensional marts for BI, and wide tables for the heaviest dashboards and ML features. The deciding factors are source volatility, audit need and who reads the layer. See modeling on the platform.

Takeaway

The model is chosen per layer. Raw is source-shaped, silver is conformed, and gold takes a dimensional, vault-backed or wide shape depending on its reader.
ML features need point-in-time correctness and one definition for training and serving, or the model learns from the future and skews in production.
Additive changes are safe and breaking changes are not. Backward compatibility is the analytics default, with consumers upgraded first.
Registries catch structural breaks only. A change in meaning passes the check and corrupts metrics, which is the gap a data contract closes.
A contract covers schema, semantics and service levels, and it is enforced in the producer's CI and at ingest.
A catalog with lineage and owners answers what breaks if this changes and where a number came from. Access policy on classification tags protects new PII automatically.
Erasure on an immutable lake needs row-level deletes or crypto-shredding plus lineage. Cost needs showback, right-sizing and design-time pruning.

The pieces are now in place, so put them together on a full scenario. → The Design Session