Analytics Without Surveillance

The question the whole loop is really testing

Strip away the small talk and the SQL screen and the culture-fit chatter, and this loop has one load-bearing question in it: "How would you build analytics here, given we don't log prompts?" Everything else — the metrics chapter, the stack chapter, the on-chain chapter — is downstream of how you handle this. The company — a privacy-first, uncensored consumer-AI platform — is hiring its first serious analytics person precisely because someone has to answer this for real, in production, not as a whiteboard hypothetical. If your answer is good, the rest of the loop is you confirming a decision they've already started to make.

The trap is to hear the question as a complaint — "we're hobbled, help us cope" — and respond with workarounds and apologies. That framing loses. The candidate who wins does a hard reframe in the first ten seconds:

"Flying on instruments" is the exact right metaphor and worth unpacking, because it does real work. A pilot in cloud can't see the horizon — and flies precisely, safely, at scale, every day, on a panel of aggregate signals: altitude, airspeed, attitude, heading. Nobody calls that flying blind. The company's data function is the same posture: you give up the direct view (prompt content, per-user trails) and you build a disciplined instrument panel out of everything that doesn't require surveillance — billing events, aggregate usage metadata, content-free product events, fully-public on-chain token data. Done right, that panel answers essentially every strategic business question leadership actually asks, because those questions are almost never "what is user #48213 typing" — they're "is free-to-paid conversion improving," "which model tier drives retention," "what's our inference margin trending toward."

The stakes are worth naming explicitly in the room. The company's posting doesn't just tolerate this stance — it makes protecting it part of the job: you're asked to "advocate against unnecessary data collection to protect privacy commitments." So this isn't a technical corner you're being asked to accept; it's a principle you're being hired to guard, sometimes against your own colleagues who'd love a per-user funnel. A senior candidate treats that as the interesting part of the role, not the annoying part. The rest of this chapter is the machinery that makes "flying on instruments" a real, defensible, ship-it plan rather than a nice slogan.

The safe-vs-unsafe data map

The single most useful artifact to carry into this interview is a crisp map of what you can and cannot touch. Most candidates gesture vaguely at "aggregate data." You want to be able to enumerate the sources, name why each is safe or off-limits, and never fumble the line. Here it is as a two-column map — memorize the shape of it.

What you lose, honestly

Here is where good candidates separate from great ones. A merely-good candidate sells the reframe so hard they imply nothing is lost. A great candidate — and a great hire, since "be transparent about analytical uncertainty" is written into the job — names the real costs plainly, because the interviewer knows them and is testing whether you do too. Pretending the constraint is free reads as either naïveté or spin. So say it straight:

Notice the shape of the compensation column: it's not "we recover everything," it's "we recover the decision-relevant part through aggregation, opt-in, and qualitative triangulation, and we accept honestly-wider error bars on the rest." That's a senior posture. It also sets up a natural, credible answer to the follow-up they love — "so what can't you do?" — where you calmly list these instead of getting defensive.

The privacy-preserving toolkit

The reframe gets you in the door; the toolkit proves you can actually deliver. These are the named techniques a senior privacy-aware analyst reaches for. Know what each does, when to use it, and — for the two or three that matter most — be able to sketch the code. They stack, roughly, from cheapest-and-most-foundational to most-specialized.

1. Data minimization & purpose-bound schemas (the foundation)

The single most important technique isn't fancy math — it's not collecting the data in the first place. The classic mistake is "collect everything, anonymize later." Anonymize-after-the-fact fails: raw data leaks, joins re-identify, and "we'll scrub it" never fully happens. The correct posture is purpose-bound collection — every event has a declared purpose, a defined granularity, and a retention window, and you literally cannot emit a field that isn't on the schema. This is the technique the job description is pointing at when it says "advocate against unnecessary data collection." You are the person who owns this schema and says no.

# Purpose-bound event schema. If a field isn't declared here with a
# purpose + granularity + retention, the collector rejects it.
- event: image_generated
  purpose: product_analytics          # why we may collect this at all
  pii: none                           # enforced: no raw content, no user text
  granularity: cohort                 # never queried per-identified-user
  retention_days: 395                 # auto-expire; not "keep forever"
  properties:
    model_family:   {type: enum, values: [flux, sd, other]}  # not the prompt
    tier:           {type: enum, values: [free, pro, pro_plus, max]}
    latency_bucket: {type: enum, values: ["<2s", "2-5s", "5-15s", ">15s"]}  # bucketed, not raw ms
    ts_hour:        {type: timestamp, truncate: hour}         # coarsened time
  forbidden:                          # documented, enforced denials
    - prompt_text
    - image_bytes
    - user_ip
    - device_fingerprint

Walking an interviewer through a schema like this is disproportionately convincing, because it makes the abstract concrete: privacy isn't a policy PDF, it's a forbidden block the pipeline enforces. It also demonstrates you'd build the guardrail into the system so a well-meaning PM can't accidentally start logging prompt text.

2. Aggregation over granularity

Where minimization is about not collecting, aggregation is about not storing at row-per-event granularity. Roll up into time windows (hourly/daily), and for the two counts that usually tempt people back toward per-user rows — distinct users and event frequencies — use probabilistic sketches that give you the metric without keeping the individuals:

• HyperLogLog for distinct counts (DAU, distinct models used) — a few KB estimates cardinality to ~1-2% error, and critically it stores no user IDs, only a register of hashed maxima. You get "how many distinct users" without a list of who.
• Count-Min sketch for frequency ("how often is each model invoked") — again, a fixed-size structure that never materializes per-user rows.

-- Postgres has HLL via the postgresql-hll extension.
-- Store ONE sketch per day per tier — no per-user rows survive.
CREATE TABLE dau_sketch (
  day        date,
  tier       text,
  users_hll  hll          -- the sketch, not a user list
);

-- Ingest: fold hashed user keys into the daily sketch, then drop the raw key.
INSERT INTO dau_sketch (day, tier, users_hll)
SELECT date_trunc('day', ts)::date, tier,
       hll_add_agg(hll_hash_text(user_key))   -- hashed, then discarded
FROM   raw_events
GROUP  BY 1, 2;

-- Query distinct actives with NO ability to enumerate individuals:
SELECT day, tier, hll_cardinality(users_hll)::int AS dau
FROM   dau_sketch
ORDER  BY day;

The point to make out loud: an HLL sketch is structurally incapable of answering "who was active" — it can only answer "how many." That's privacy enforced by the data structure itself, which is stronger than privacy enforced by a policy you promise to follow.

3. k-anonymity / suppress small cells

The workhorse you'll use every day. Rule: never report a cohort of fewer than k users (k = 20 or 50 is typical). A "cohort" of 1-3 users is just a person with extra steps — small cells are how "aggregate" reports leak individuals. Bake the threshold into the query and the BI layer so it's not a thing anyone can forget:

-- Model adoption by country + tier, suppressing any cell under k=50.
-- A cohort is only reported if enough distinct users share it.
WITH cohorts AS (
  SELECT country, tier, model_family,
         COUNT(DISTINCT user_key) AS n_users,
         SUM(requests)            AS total_requests
  FROM   usage_daily
  WHERE  day >= current_date - 30
  GROUP  BY country, tier, model_family
)
SELECT country, tier, model_family, n_users, total_requests
FROM   cohorts
WHERE  n_users >= 50            -- k-anonymity threshold: suppress small cells
ORDER  BY total_requests DESC;

4. Differential privacy (for anything shared externally)

When an aggregate leaves the trust boundary — a public transparency report, a blog stat, a number handed to a partner — k-anonymity isn't always enough, because repeated or overlapping queries can triangulate. Differential privacy adds calibrated noise so any single individual's presence or absence can't be inferred, and it's governed by an epsilon budget (smaller ε = more privacy, more noise) that you spend down across queries. Use it selectively — for externally-shared or high-sensitivity aggregates — not on every internal dashboard, where it just adds noise you don't need.

import numpy as np

def dp_count(true_count: int, epsilon: float = 1.0) -> int:
    """Laplace mechanism for a counting query (sensitivity = 1).
    Publishable count where one user's in/out is statistically hidden."""
    scale = 1.0 / epsilon              # noise grows as epsilon shrinks
    noisy = true_count + np.random.laplace(loc=0.0, scale=scale)
    return max(0, round(noisy))

# Externally-published stat, e.g. a transparency report:
published = dp_count(true_count=48213, epsilon=0.5)   # ~48213 +/- a little noise
# Track spend: each release debits a global epsilon budget so repeated
# queries can't be averaged to strip the noise.

5. Local DP / randomized response

The stronger cousin: instead of trusting the server to add noise, the client perturbs the data before it ever leaves the device — so even a compromised server never sees a true individual value. The textbook primitive is randomized response (answer truthfully with probability p, flip a coin otherwise); at population scale the aggregate is recoverable while each individual answer is deniable. This is exactly the Apple model (below) and the right tool for opt-in telemetry where you want a signal but refuse to be trusted with the raw value.

6. Tokenization — per-purpose HMAC IDs

When you do need to join two datasets (say, consented wallet ↔ account for staked-API access, per Chapter 06), never use the raw user ID as the key across purposes — that's how systems accidentally build the cross-context identity graph you're trying to avoid. Derive a per-purpose pseudonym with a keyed HMAC, so the same user has different, unlinkable IDs in different contexts, and no ID reverses to the account without the secret key:

import hmac, hashlib

def pseudonym(user_id: str, purpose: str, secret: bytes) -> str:
    """Per-purpose HMAC id. Same user -> different id per purpose,
    so datasets can't be cross-joined into one identity graph."""
    msg = f"{purpose}:{user_id}".encode()
    return hmac.new(secret, msg, hashlib.sha256).hexdigest()

# Billing analytics and product analytics get UNLINKABLE keys:
pseudonym("u_123", "billing",  SECRET)   # -> 9f2a...
pseudonym("u_123", "product",  SECRET)   # -> c471...  (can't be joined to billing)

7. Client-side event batching & self-hosting

Two structural moves that reduce exposure without any statistics. Client-side batching — buffering events and sending them in coarse batches — blunts timing side-channels (exact per-action timestamps are themselves a fingerprint). Self-hosting the analytics stack (a theme of Chapter 03) keeps every one of these datasets inside the company's own trust boundary, so you're not shipping even metadata to a third party — which is both the privacy-honest choice and the culturally-aligned one at a company built on not trusting third parties with your data.

PostHog, configured for privacy

The company already names PostHog as its product-analytics tool in the stack, so this is the one concrete config you should be able to speak to without notes — it signals you've thought about the actual tools they run, not a generic plan. PostHog defaults are built for typical product teams that want rich person-level profiles; the whole job here is to lock those defaults down so the tool respects the same no-surveillance contract as everything else.

{
  "person_profiles": "identified_only",   // no auto-profile on anonymous visitors
  "autocapture": false,                    // no blanket DOM capture; declare events explicitly
  "capture_pageview": false,               // pageviews can leak content in URLs/titles
  "ip": false,                             // do not store IP against events
  "property_denylist": ["$ip", "email", "$current_url", "prompt", "message"],
  "mask_all_text": true,                   // session-replay text masking, if replay is on at all
  "mask_all_element_attributes": true,
  "sanitize_properties": "strip_pii",      // scrub before events leave the client
  "self_hosted": true                      // run PostHog in our own trust boundary
}

• person_profiles: 'identified_only' — the key switch: PostHog won't build persistent profiles for anonymous users, so casual visitors never become tracked identities.
• PII masking & ph-no-capture — add the ph-no-capture CSS class to any element (the chat box above all) so its contents are never captured in autocapture or session replay; deny-list prompt, message, email, $ip at the property level.
• IP & email anonymization — disable IP storage and never send email as a property; identify on an opaque per-purpose key, not a real address.
• Self-hosting — PostHog is open-source and self-hostable; running it inside the company's infrastructure means even content-free product events never leave the trust boundary. Culturally, this is the obvious choice.

Proof it works at scale

An interviewer's natural pushback is "that's a nice theory, but does privacy-preserving analytics actually work at scale, or is it academic?" Have two real, large-scale precedents ready — being able to cite them turns your answer from opinion into established practice, and shows you know the field beyond this one company.

The rhetorical value is that both are companies whose brand is privacy — the same position the company occupies — proving the constraint and a real data/experimentation practice coexist at scale. When you name them, you're implicitly saying: "this is a solved posture at billion-user companies; I'm proposing to bring that discipline here, sized for a startup of roughly 50 people."

The six-beat answer to the core question

When they ask "how would you build analytics here, given we don't log prompts?" — and they will — don't improvise. Deliver a structured answer in six beats. It's memorizable, it hits every point above in order, and its very structure signals that you've thought about this as a system, not a vibe. Practice it until you can walk it in ninety seconds and then go deep on whichever beat they probe.

1. Reframe the constraint as the product. "First — I don't see the no-logging rule as a limitation to work around. It's the company's core differentiator, and my job includes defending it. We're not flying blind; we're flying on instruments." (Sets alignment before tactics.)
2. Define a content-free north star. Anchor on a metric that needs zero user content — e.g. gross-margin-adjusted net revenue retention, or free-to-paid conversion by signup cohort. "Everything I build serves a metric I can compute without ever reading a prompt." (Shows business sense; see Chapter 04.)
3. Map safe vs unsafe data. Enumerate the sources: billing/Stripe, aggregate usage metadata, content-free PostHog events, public on-chain token data ($TOKEN/$CREDIT) — and name what's off-limits and why (content, per-user trails, identity graphs, fingerprints). (This is the map from §2.)
4. Instrument aggregate / anonymous events with the toolkit. "I collect via purpose-bound schemas — minimize first, never anonymize-after — report cohorts with k-anonymity, use HyperLogLog for distinct counts, and differential privacy for anything we publish externally." (The §4 toolkit, named crisply.)
5. Build a lean, self-hosted stack. "Consolidate those sources into a warehouse via Airbyte/custom loaders, model with dbt into financial/product/growth marts, surface in self-hosted Metabase — all inside our trust boundary." (Hand off to Chapter 03.)
6. Establish culture, guardrails, and a 30-60-90. "Then I make the guardrails structural — the schema enforces privacy, small cells auto-suppress — evangelize cohort-thinking across the team, and sequence it: audit & foundations first ~30 days, first dashboards by 60, a v1 data model and metric definitions by 90." (The privacy advocacy clause, made operational; see Chapter 10.)