Governance, Security & Compliance

Governance as an enabler, not a blocker

The default mental model of "data governance" is a compliance officer with a clipboard who slows everything down. That model is backwards for a platform team. Everything we built toward in chapter 09 — self-serve analytics for tens to low hundreds of internal users — and everything we will build in chapter 12 and chapter 13 — features, training data, and an LLM answering questions in Slack — only works if governance is good enough that you can open the doors.

The balance is the whole game. There are two failure modes, and they are symmetric:

• Too loose — and you leak. A growth analyst's exploratory query touches raw KYC identity documents; an LLM agent retrieves a customer's crypto wallet address into a Slack thread; one provider's payout data shows up in another provider's dashboard. Each of these is a breach, a regulatory event, and a trust collapse for a company whose product is trust.
• Too tight — and you ossify. Everything routes through a ticket queue to the one person with the keys. Self-serve dies, shadow data pipelines bloom in spreadsheets and personal laptops (which is worse for security), and the AI initiatives stall because no one will let the model near the data.

This reframing has a concrete consequence: governance must be declarative, attached to the data, and enforced by the engine — not a set of conventions in a wiki that analysts are trusted to follow. The rest of the chapter is about how to express and enforce policy so that the platform can default to open.

Access control: the model that authorizes a query

Access control answers one question billions of times a day: is this principal allowed to read this data? The two dominant models are RBAC (role-based) and ABAC (attribute-based), and a mature platform uses both — RBAC for the coarse grant, ABAC for the fine-grained policy.

The practical pattern: RBAC gets you into the room; ABAC decides which seats you can sit in. A finance_analyst role grants access to the billing domain (RBAC), but an attribute policy still masks the customer's tax ID inside it unless that analyst additionally carries a restricted_pii attribute (ABAC). You get a small number of human-legible roles and fine control without role explosion.

Principals, roles, grants

The model is a chain. Principals (humans, service accounts, the AI agent) are assigned roles; roles hold grants on catalog objects; and on top of the grant sit row- and column-level policies that the engine evaluates at query time. Crucially, roles map to the data domains from Start Here — domains are the seam for access control, exactly as promised there.

Two principles keep this from rotting:

• Least privilege by default. A new principal starts with access to nothing sensitive — the public and internal tiers only. Elevated access is requested, time-boxed where possible, and granted to a role, never to an individual. This is what lets you onboard fast: open data is open by default, sensitive data requires an explicit, audited step.
• Grant on objects, enforce in the engine. The grant lives on a catalog object (a table, a view, a domain namespace), and the policy is evaluated by the query engine on every read. There is no path — not a dashboard, not a notebook, not the AI agent, not a CSV export — that bypasses the policy, because the policy lives below all of them at the engine boundary.

How a query gets authorized

When alice (a growth_analyst) runs SELECT * FROM marketplace.fct_rental, the engine: (1) resolves her principal to her active roles; (2) checks that a role holds a SELECT grant on the object — deny if not; (3) rewrites the query to apply any row-level policy (e.g. only rows in her permitted regions); (4) applies any column-level policy (mask or null out columns whose tag she lacks clearance for); (5) executes, and (6) writes an audit record of the access. The analyst sees a clean result set and never knows a policy fired — which is the point.

PII & KYC handling: minimizing blast radius

GridDP's most sensitive data comes from source 8 — payments: the KYC/AML provider emits identity-verification outcomes and the documents behind them; the crypto rails carry wallet addresses; Stripe carries card and bank details. This is the data a regulator, an attacker, and a journalist all care about most. The governing discipline is to classify everything, then minimize the blast radius of the sensitive slice.

The key move for the Restricted tier is that raw values should ideally never land in the lake at all. We tokenize on ingest: the pipeline replaces the sensitive value with a non-reversible (or vault-reversible) token at the moment it enters the platform, and the raw value lives only in a dedicated, separately-controlled vault. Analysts join and count on the token; almost no one ever needs the raw value, and the few who do (a T&S investigator following a fraud ring across wallets) go through an audited de-tokenization path.

Layered on top of tokenization are two more controls, applied in priority order — and this is the order a senior reviewer will check them in:

1. Minimize at the source. The cheapest PII to govern is the PII you never ingest. Following the data-contract discipline from chapter 01, the platform refuses to load fields it doesn't need — a behavioral event does not need to carry an email; flag it in the contract (pii: false is an assertion the platform checks).
2. Tokenize / pseudonymize on ingest. What must be ingested for joins (wallet, tax ID) lands as a token, not a value, as above.
3. Mask in queries. The handful of confidential columns that do live in the lake (e.g. a partial card suffix for support) are masked by a column-level policy keyed on a classification tag, so the masking applies everywhere the column is read — view, dashboard, or AI.
4. Restricted domains. The KYC and raw-payments tables live in a separate restricted namespace with its own role, so even getting near them is an explicit, audited grant.

-- 1. Classify the column once, in the catalog. Policy follows the tag,
--    not the column name, so it applies everywhere the column appears.
ALTER TABLE billing.dim_customer
  ALTER COLUMN tax_id        SET TAG classification = 'restricted';
ALTER TABLE billing.dim_customer
  ALTER COLUMN card_last4    SET TAG classification = 'confidential';

-- 2. One masking policy, applied to a tag, covers thousands of columns.
CREATE MASKING POLICY mask_by_classification AS (val STRING) RETURNS STRING ->
  CASE
    WHEN is_role_in_session('restricted_pii')              THEN val          -- full reveal
    WHEN is_role_in_session('finance_analyst')             THEN 'XXXX'       -- redacted
    ELSE NULL                                                                 -- invisible
  END;

-- 3. Bind the policy to the tag. New restricted columns are protected on creation.
ALTER TAG classification SET MASKING POLICY mask_by_classification
  FOR COLUMNS WITH TAG classification IN ('restricted','confidential');

The marketplace isolation twist

Here is the wrinkle that makes a marketplace harder than a typical SaaS company. GridDP holds data about both sides of the market: providers (GPU owners) and clients (renters), per the two-sided spine in Start Here. Governance must guarantee that internal access never leaks one customer's data to another, nor to the wrong internal team.

The risks are not hypothetical:

• A supply-ops analyst studying provider economics should see provider payouts and fleet behavior — but must not be able to pull a client's rental history or spend.
• A growth analyst optimizing client funnels should see client behavior — but not a competing provider's machine inventory and pricing strategy, which is commercially sensitive to that provider.
• No analyst building a dashboard for Provider A should be able to construct a view that exposes Provider B's utilization or earnings.

The mechanism is multi-tenant row-level security: a single physical table (e.g. supply.fct_provider_economics) is sliced by a row-level policy that scopes each principal to the rows their role legitimately covers. Internal analysts get a cross-tenant view only when their role explicitly authorizes aggregate analytics, and even then the policy can force aggregation (no single-provider drill-down) for roles that should only see the market in the large.

-- Row policy on a shared provider-facing table. Evaluated on EVERY read.
CREATE ROW ACCESS POLICY provider_scope AS (provider_id STRING) RETURNS BOOLEAN ->
  CASE
    -- Cross-tenant aggregate analytics: market-wide, no single-provider rows
    WHEN is_role_in_session('supply_ops')        THEN TRUE
    -- A provider self-serve seat sees ONLY its own rows (the hard tenant wall)
    WHEN is_role_in_session('provider_portal')   THEN provider_id = current_tenant_id()
    -- Demand-side roles have no business in supply-side provider data
    ELSE FALSE
  END;

ALTER TABLE supply.fct_provider_economics
  ADD ROW ACCESS POLICY provider_scope ON (provider_id);

The same pattern, keyed on account_id, walls off client-side tables from supply roles. The win: tenant isolation is a property of the data, enforced once, rather than a discipline every dashboard author must remember.

Data residency & export controls

This section is where GridDP's specific reality bites hardest, and it's a topic most generic data-platform guides skip entirely. A GPU marketplace is directly exposed to chip export-control regimes. US export controls restrict who may access high-end accelerators and from which countries; a GPU marketplace's typical operating posture involves filtering by host country and screening access to controlled hardware. That regulatory exposure flows straight into the data platform, because the platform is where you prove the controls were honored.

Two distinct concerns, often muddled:

• Data residency — where the bytes physically live. A client in the EU has GDPR expectations; some jurisdictions require that personal data not leave the region. The platform must be able to model and enforce regional storage, so EU-subject PII lands in EU-region object storage and is queried by EU-region compute.
• Export controls — which hardware may serve which counterparties. The marketplace must not match a controlled GPU to a client in a sanctioned/embargoed jurisdiction, and the data platform must retain the host-country and screening attributes needed to demonstrate compliance after the fact.

Because the marketplace spans many countries on both sides, this is not abstract: a single rental can pair a provider host in one jurisdiction with a client in another, and both the match logic (upstream) and the audit record (in the platform) must capture the country attributes. The platform models where data lives and where workloads run as first-class governed attributes.

Audit & compliance

Access control decides what can happen; audit proves what did. For a company holding KYC data and exposed to export controls, the audit trail is not bureaucratic overhead — it is the evidence that satisfies a SOC 2 auditor, a GDPR regulator, and an export-control review.

• Access logging. Every read of Confidential or Restricted data writes an immutable audit record: principal, role, object, columns, row count, query text, timestamp, source region. The de-tokenization vault logs every reveal. "Who queried KYC last quarter?" must be a one-query answer — and the answer itself goes to a restricted log only the security team reads.
• SOC 2. Most of the controls in this chapter are the SOC 2 evidence: least-privilege provisioning, periodic access reviews (recertify role membership), separation of duties, and the audit log itself. Build them as platform features and the audit becomes a query, not a fire drill.
• GDPR & right-to-be-forgotten. A deletion request must propagate across the lake. With tokenization, the cheapest implementation is crypto-shredding: destroy the subject's key in the vault and every token referencing them becomes permanently unresolvable — the personal data is effectively erased everywhere at once, without rewriting petabytes of immutable lake files. Where raw values do exist, schedule hard deletes (Iceberg/Delta row-level deletes, then compaction) on the relevant partitions.
• Retention policies. Different classes have different clocks: behavioral events may be kept long in aggregate but pruned at row-grain after N months; KYC docs are kept exactly as long as AML regulation requires and no longer (keeping PII past its purpose is itself a liability). Retention is declared as policy on the catalog object and enforced by a scheduled job.

Cost governance / FinOps for data

Governance is not only about access — it is also about spend. Recall the scale numbers: ~150 M telemetry samples a day and 50–90 TB/yr of growth. An open self-serve platform without cost guardrails is a way to discover, via an invoice, that an analyst left a SELECT * against the raw firehose running over the weekend. FinOps is the governance discipline that keeps openness affordable.

The core idea mirrors access control: make cost visible and attributable, then put automated guardrails on it. Chargeback/showback assigns each query's cost back to the team that ran it — showback (visibility only) changes behavior surprisingly well; chargeback (actual budget) is the escalation when it doesn't.

Governance per stack

Every concept above — classification tags, masking, row policies, lineage, audit — has a concrete home in each of the three reference stacks from Start Here. The capabilities converge; the seams and the lock-in differ.

The strategic note from chapter 03 holds here too: Stack A buys portability and fine-grained control (OPA can express any policy) at the cost of operating the governance plane yourself; B and C give you a coherent, audited governance surface out of the box in exchange for lock-in. For a marketplace with real regulatory exposure, a turnkey, well-audited governance layer is often worth the lock-in — the cost of a governance gap is measured in breaches, not credits.

Takeaway

• Govern to enable: good, declarative, engine-enforced policy is what lets the platform default to open for self-serve (ch. 09) and AI (ch. 13). Too loose leaks; too tight ossifies into shadow pipelines.
• Use RBAC to get into the room, ABAC to choose the seat: domain roles for coarse access, attribute policies (tags, region, tenant) for row- and column-level control without role explosion.
• For KYC/crypto/payment PII, minimize, then tokenize on ingest so raw values never land in the lake; mask the rest by classification tag; fence restricted domains. This also makes GDPR erasure a key-delete (crypto-shredding).
• A marketplace holds both providers' and clients' data — multi-tenant row-level security must wall tenant-from-tenant (a breach if it fails) and scope internal teams (hygiene if it fails).
• GridDP is really exposed to chip export controls and residency: model where bytes and workloads live as governed attributes, enforced through the same ABAC layer.
• Audit proves what happened (SOC 2, GDPR); FinOps governs the firehose's cost — and the cheap path and the safe path are designed to be the same path.

With trustworthy data now governed and safely served, we turn to the consumer that pushes the platform hardest: machine learning. Next → the platform for AI/ML — features, training data, and offline↔online parity.