Experimentation for Rapid Iteration

Why experimentation is hard here

Textbook A/B testing assumes three things the company deliberately breaks. It assumes a stable user identity you can randomize on and follow over time. It assumes you can join a user's assignment to their downstream behavior at the individual level. And it assumes a product surface that changes slowly enough that a metric measured in March still means the same thing in June. The company violates all three at once, and a senior candidate should be able to name why before proposing a single test.

• No user-level identity graph. Prompts are encrypted client-side and never logged; chat history lives in the browser's IndexedDB and does not sync across devices. There is no durable per-user behavioral trail to attach an experiment arm to. A logged-out user on Firefox this morning and Safari tonight is, to your analytics, two different entities — by design.
• No cross-device tracking. Because state is browser-local, the same human is deliberately unstitchable across devices and sessions. That is a privacy feature and a randomization headache in one.
• Fast product cadence. The company ships weekly. A ~50-person, high-autonomy team that deeply respects people who ship will not wait two weeks for a clean readout, and the surface under test may itself have moved by the time the test ends.

Standing up experiment tracking

"Experiment-tracking infrastructure" is a deliverable, not a vibe. Break it into the four components every experimentation platform must provide, then map each to a tool the company can actually run inside its own trust boundary (this ties directly to Chapter 03, where the same self-hosting logic drove the stack).

Start with PostHog — it's already chosen for product analytics, and its feature-flag + experiments module gives you assignment, exposure, and a results surface with essentially zero new infrastructure. Configure it privacy-first: person_profiles: 'identified_only', IP anonymization on, PII masking, and self-host if the volume justifies it, so raw event data stays inside your boundary.

Add GrowthBook when experimentation becomes core. GrowthBook is open-source and warehouse-native: it runs its statistics on the tables already in your own warehouse rather than shipping event data to a vendor. For a privacy-first company that is a genuine architectural advantage, not just a cost story — the experiment analysis never leaves your trust boundary, and you define metrics once in SQL/dbt and reuse them (which is exactly what makes them robust to iteration; see below). Statsig is the heavyweight option if you reach the scale where you need sequential testing, CUPED, and stratification out of the box — but it's overkill for a first hire and pulls data toward a vendor.

Privacy-safe A/B design

The crux question: how do you randomize without a persistent user ID? The tension is real — randomization wants a stable key, and the whole product is built to avoid stable per-user keys. Here is how you resolve it rather than hand-wave it.

Choosing a randomization unit

• Session- or device-bucketed assignment. Hash a content-free, browser-local identifier (or a random bucket assigned on first load and stored in IndexedDB alongside the app state that's already there) into N buckets, then map buckets to arms. This is anonymous — the identifier carries no personal information and never leaves as a user profile. It's exactly the pattern DuckDuckGo uses with its atb ("add-to-browser") cohort token: a random, non-identifying string that enables A/B measurement without tracking a person.
• Server-side assignment on a consented key. For logged-in / subscription surfaces, you already legitimately hold a billing account ID (Stripe). Hashing that server-side for assignment is fine — it's a first-party, consented relationship, not a tracking identifier reconstructed from behavior.
• Cluster/geo randomization. When even a device bucket is too much identity, randomize at a coarser level (rollout by region, by cohort, or by time-boxed holdout) and measure the aggregate.

Measuring on aggregates, not individuals

The measurement side is where privacy-safe design really diverges from the textbook. You do not follow individuals from assignment to outcome. You compare cohort-level aggregates: conversion rate of bucket A vs bucket B, tokens-per-session in arm A vs arm B, day-2 return rate of the device cohort exposed to the new onboarding vs the old. The unit of analysis is the arm, not the person.

Always ship guardrail metrics alongside the primary: error rate, latency, and — critically for the company — a privacy guardrail (no growth in identifiable data captured). Guardrails are how you move fast without the experiment quietly breaking something you weren't watching.

The design discipline

The single most common senior-level mistake in an experimentation interview is jumping to statistics before nailing the design. A candidate who opens with "I'd run a two-sample t-test at alpha 0.05" has skipped the part that actually matters. Given a fuzzy product question — "does the new model-picker improve engagement?" — the discipline is to convert it into a specified experiment first:

1. Hypothesis. A directional, falsifiable statement: "Surfacing recommended models on first chat increases the share of new sessions that send a second prompt." Not "the picker is better."
2. Primary metric. One metric that decides the call — here, second-prompt rate per session. Resist a basket of five co-primaries; pick the one you'd ship on.
3. Guardrail metrics. What must not get worse: latency, error rate, upgrade-click rate, and the privacy guardrail.
4. Randomization unit. Session bucket or device bucket (per above), chosen so the unit matches the decision and respects the no-tracking rule.
5. Sample size & power. Compute the N needed to detect the smallest effect worth shipping (the MDE) at your power target — before launch, so you know if the test is even feasible given traffic.
6. Runtime. Long enough to cover at least one full weekly cycle (weekday/weekend usage differs), capped so it doesn't block the roadmap.
7. Stopping rule. Decided in advance: fixed horizon, or an explicit sequential procedure. Never "we'll look daily and stop when it's significant."

Stats you must defend

Keep the statistics applied — every concept should translate to a product decision, not a lecture. The bar for a senior hire is that you can defend these under questioning and know when each matters.

• Power & sample size. Power is the probability of detecting a real effect of a given size. Compute N up front from your MDE, baseline rate, alpha, and power; if the required N exceeds realistic traffic in a reasonable runtime, the honest move is to say the test can't be powered and propose a bigger MDE or a different approach.
• Statistical vs practical significance. With a large user base you can make a 0.1% lift "significant" and utterly irrelevant. Always report the effect size and a confidence interval, and pre-declare the effect worth shipping. This is the one most people get wrong; leading with it reads as operator, not textbook.
• p-values, Type I / Type II. Type I = shipping a change that does nothing (false positive); Type II = killing a change that actually helped (false negative). Which is costlier depends on the decision — for an irreversible, brand-adjacent change, weight Type I harder.
• Sequential testing & peeking. If the team wants to monitor continuously (they will), use a method built for it — always-valid p-values / group-sequential boundaries — instead of pretending a fixed-horizon test can be peeked at safely.
• CUPED (variance reduction). Uses a pre-experiment covariate to strip out baseline variance, often cutting required sample size 20–50%. This is especially valuable here, where anonymous, aggregate measurement means samples are scarcer and noisier than at a track-everything company. Naming CUPED as the lever for "we can't get enough sample" is a strong senior signal.

The sample-size calculation you should be able to sketch live:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Product question: does the new model-picker lift the "second prompt" rate?
baseline = 0.40        # current share of new sessions sending a 2nd prompt
mde_abs  = 0.02        # smallest lift worth shipping: 40% -> 42%
alpha    = 0.05        # tolerated false-positive rate (Type I)
power    = 0.80        # probability of detecting a true effect of mde size

# Cohen's h effect size for two proportions
effect = proportion_effectsize(baseline + mde_abs, baseline)

n_per_arm = NormalIndPower().solve_power(
    effect_size=effect, alpha=alpha, power=power, ratio=1, alternative="two-sided"
)
print(f"Need ~{n_per_arm:,.0f} sessions per arm")

# Feasibility: at ~X qualifying sessions/day, runtime = 2 * n_per_arm / X.
# If that exceeds a sane runtime, raise the MDE or apply CUPED to cut N.
# Randomize on an anonymous device/session bucket -- never a durable user ID.

And the aggregate readout, computed in SQL with a k-anonymity floor so no small cell can single anyone out — the same suppress-small-cells discipline from Chapter 02:

-- Primary metric by variant, measured on anonymous session buckets.
-- No user-level join anywhere; the unit of analysis is the arm.
with exposed as (
    select
        variant,                      -- 'control' | 'treatment'
        session_bucket,               -- content-free hashed bucket, not a user id
        max(sent_second_prompt) as converted   -- 1 if the session hit the goal
    from experiment_exposures
    where experiment_key = 'model_picker_v3'
      and exposed_at >= '2026-08-01'            -- full weekly cycles only
    group by variant, session_bucket
)
select
    variant,
    count(*)                                   as sessions,
    sum(converted)                             as conversions,
    round(avg(converted)::numeric, 4)          as conversion_rate,
    -- Wald 95% CI; report the interval, never a bare point estimate
    round(1.96 * sqrt(avg(converted)*(1-avg(converted)) / count(*))::numeric, 4) as ci_halfwidth
from exposed
group by variant
having count(*) >= 100        -- k-anonymity floor: suppress under-powered / tiny cells
order by variant;

Robust to rapid iteration

The posting's phrase "metrics robust to the company's rapid product iteration" is a direct requirement, and experimentation is where it bites hardest. If the definition of "activated" or "engaged session" drifts every time the product ships, then two experiments run a month apart aren't comparable, your holdouts are meaningless, and every readout is quietly measuring a moving target. The senior job is to keep metric definitions stable even while the product changes weekly.

• A versioned metric layer. Define primary and guardrail metrics once, in dbt / the GrowthBook metric library, under version control — not ad hoc in each experiment. When a definition genuinely must change, bump the version and annotate the break, so anyone reading an old result knows which definition produced it. This is the single most important mechanism for comparability across a fast-moving roadmap, and it's why the warehouse-native tooling in section 2 matters.
• Guardrails that outlive any one test. A standing set of guardrails (latency, error rate, upgrade rate, privacy guardrail) applied to every experiment means the definition of "did we break something" doesn't get renegotiated per launch.
• Long-run holdouts. Keep a small, persistent holdout that doesn't get the stream of weekly changes, so you can measure the cumulative impact of rapid iteration — not just each change in isolation. It's the antidote to shipping ten individually-positive changes that sum to a regression.