Experimentation for Rapid Iteration

Why experimentation is hard here

Textbook A/B testing assumes three things the company deliberately breaks. It assumes a stable user identity you can randomize on and follow over time. It assumes you can join a user's assignment to their downstream behavior at the individual level. And it assumes a product surface that changes slowly enough that a metric measured in March still means the same thing in June. The company violates all three at once, and a senior candidate should be able to name why before proposing a single test.

• No user-level identity graph. Prompts are encrypted client-side and never logged; chat history lives in the browser's IndexedDB and does not sync across devices. There is no durable per-user behavioral trail to attach an experiment arm to. A logged-out user on Firefox this morning and Safari tonight is, to your analytics, two different entities — by design.
• No cross-device tracking. Because state is browser-local, the same human is deliberately unstitchable across devices and sessions. That is a privacy feature and a randomization headache in one.
• Fast product cadence. The company ships weekly. A ~50-person, high-autonomy team that deeply respects people who ship will not wait two weeks for a clean readout, and the surface under test may itself have moved by the time the test ends.

Standing up experiment tracking

"Experiment-tracking infrastructure" is a deliverable, not a vibe. Break it into the four components every experimentation platform must provide, then map each to a tool the company can actually run inside its own trust boundary (this ties directly to Chapter 03, where the same self-hosting logic drove the stack).

Start with PostHog — it's already chosen for product analytics, and its feature-flag + experiments module gives you assignment, exposure, and a results surface with essentially zero new infrastructure. Configure it privacy-first: person_profiles: 'identified_only', IP anonymization on, PII masking, and self-host if the volume justifies it, so raw event data stays inside your boundary.

Add GrowthBook when experimentation becomes core. GrowthBook is open-source and warehouse-native: it runs its statistics on the tables already in your own warehouse rather than shipping event data to a vendor. For a privacy-first company that is a genuine architectural advantage, not just a cost story — the experiment analysis never leaves your trust boundary, and you define metrics once in SQL/dbt and reuse them (which is exactly what makes them robust to iteration; see below). Statsig is the heavyweight option if you reach the scale where you need sequential testing, CUPED, and stratification out of the box — but it's overkill for a first hire and pulls data toward a vendor.

Privacy-safe A/B design

The crux question: how do you randomize without a persistent user ID? The tension is real — randomization wants a stable key, and the whole product is built to avoid stable per-user keys. Here is how you resolve it rather than hand-wave it.

Choosing a randomization unit

• Session- or device-bucketed assignment. Hash a content-free, browser-local identifier (or a random bucket assigned on first load and stored in IndexedDB alongside the app state that's already there) into N buckets, then map buckets to arms. This is anonymous — the identifier carries no personal information and never leaves as a user profile. It's exactly the pattern DuckDuckGo uses with its atb ("anonymous A/B test") cohort token: a random, non-identifying string that enables A/B measurement without tracking a person.
• Server-side assignment on a consented key. For logged-in / subscription surfaces, you already legitimately hold a billing account ID (Stripe). Hashing that server-side for assignment is fine — it's a first-party, consented relationship, not a tracking identifier reconstructed from behavior.
• Cluster/geo randomization. When even a device bucket is too much identity, randomize at a coarser level (rollout by region, by cohort, or by time-boxed holdout) and measure the aggregate.

Measuring on aggregates, not individuals

The measurement side is where privacy-safe design really diverges from the textbook. You do not follow individuals from assignment to outcome. You compare cohort-level aggregates: conversion rate of bucket A vs bucket B, tokens-per-session in arm A vs arm B, day-2 return rate of the device cohort exposed to the new onboarding vs the old. The unit of analysis is the arm, not the person.

Always ship guardrail metrics alongside the primary: error rate, latency, and — critically for the company — a privacy guardrail (no growth in identifiable data captured). Guardrails are how you move fast without the experiment quietly breaking something you weren't watching.

The design discipline

The single most common senior-level mistake in an experimentation interview is jumping to statistics before nailing the design. A candidate who opens with "I'd run a two-sample t-test at alpha 0.05" has skipped the part that actually matters. Given a fuzzy product question — "does the new model-picker improve engagement?" — the discipline is to convert it into a specified experiment first:

1. Hypothesis. A directional, falsifiable statement: "Surfacing recommended models on first chat increases the share of new sessions that send a second prompt." Not "the picker is better."
2. Primary metric. One metric that decides the call — here, second-prompt rate per session. Resist a basket of five co-primaries; pick the one you'd ship on.
3. Guardrail metrics. What must not get worse: latency, error rate, upgrade-click rate, and the privacy guardrail.
4. Randomization unit. Session bucket or device bucket (per above), chosen so the unit matches the decision and respects the no-tracking rule.
5. Sample size & power. Compute the N needed to detect the smallest effect worth shipping (the MDE) at your power target — before launch, so you know if the test is even feasible given traffic.
6. Runtime. Long enough to cover at least one full weekly cycle (weekday/weekend usage differs), capped so it doesn't block the roadmap.
7. Stopping rule. Decided in advance: fixed horizon, or an explicit sequential procedure. Never "we'll look daily and stop when it's significant."

Stats you must defend

Keep the statistics applied — every concept should translate to a product decision, not a lecture. The bar for a senior hire is that you can defend these under questioning and know when each matters.

• Power & sample size. Power is the probability of detecting a real effect of a given size. Compute N up front from your MDE, baseline rate, alpha, and power; if the required N exceeds realistic traffic in a reasonable runtime, the honest move is to say the test can't be powered and propose a bigger MDE or a different approach.
• Statistical vs practical significance. With a large user base you can make a 0.1% lift "significant" and utterly irrelevant. Always report the effect size and a confidence interval, and pre-declare the effect worth shipping. This is the one most people get wrong; leading with it reads as operator, not textbook.
• p-values, Type I / Type II. Type I = shipping a change that does nothing (false positive); Type II = killing a change that actually helped (false negative). Which is costlier depends on the decision — for an irreversible, brand-adjacent change, weight Type I harder.
• Sequential testing & peeking. If the team wants to monitor continuously (they will), use a method built for it — always-valid p-values / group-sequential boundaries — instead of pretending a fixed-horizon test can be peeked at safely.
• CUPED (variance reduction). Uses a pre-experiment covariate to strip out baseline variance, often cutting required sample size 20–50%. This is especially valuable here, where anonymous, aggregate measurement means samples are scarcer and noisier than at a track-everything company. Naming CUPED as the lever for "we can't get enough sample" is a strong senior signal.

The sample-size calculation you should be able to sketch live:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Product question: does the new model-picker lift the "second prompt" rate?
baseline = 0.40        # current share of new sessions sending a 2nd prompt
mde_abs  = 0.02        # smallest lift worth shipping: 40% -> 42%
alpha    = 0.05        # tolerated false-positive rate (Type I)
power    = 0.80        # probability of detecting a true effect of mde size

# Cohen's h effect size for two proportions
effect = proportion_effectsize(baseline + mde_abs, baseline)

n_per_arm = NormalIndPower().solve_power(
    effect_size=effect, alpha=alpha, power=power, ratio=1, alternative="two-sided"
)
print(f"Need ~{n_per_arm:,.0f} sessions per arm")

# Feasibility: at ~X qualifying sessions/day, runtime = 2 * n_per_arm / X.
# If that exceeds a sane runtime, raise the MDE or apply CUPED to cut N.
# Randomize on an anonymous device/session bucket -- never a durable user ID.

And the aggregate readout, computed in SQL with a k-anonymity floor so no small cell can single anyone out — the same suppress-small-cells discipline from Chapter 02:

-- Primary metric by variant, measured on anonymous session buckets.
-- No user-level join anywhere; the unit of analysis is the arm.
with exposed as (
    select
        variant,                      -- 'control' | 'treatment'
        session_bucket,               -- content-free hashed bucket, not a user id
        max(sent_second_prompt) as converted   -- 1 if the session hit the goal
    from experiment_exposures
    where experiment_key = 'model_picker_v3'
      and exposed_at >= '2026-08-01'            -- full weekly cycles only
    group by variant, session_bucket
)
select
    variant,
    count(*)                                   as sessions,
    sum(converted)                             as conversions,
    round(avg(converted)::numeric, 4)          as conversion_rate,
    -- Wald 95% CI; report the interval, never a bare point estimate
    round(1.96 * sqrt(avg(converted)*(1-avg(converted)) / count(*))::numeric, 4) as ci_halfwidth
from exposed
group by variant
having count(*) >= 100        -- k-anonymity floor: suppress under-powered / tiny cells
order by variant;

Robust to rapid iteration

The posting's phrase "metrics robust to the company's rapid product iteration" is a direct requirement, and experimentation is where it bites hardest. If the definition of "activated" or "engaged session" drifts every time the product ships, then two experiments run a month apart aren't comparable, your holdouts are meaningless, and every readout is quietly measuring a moving target. The senior job is to keep metric definitions stable even while the product changes weekly.

• A versioned metric layer. Define primary and guardrail metrics once, in dbt / the GrowthBook metric library, under version control — not ad hoc in each experiment. When a definition genuinely must change, bump the version and annotate the break, so anyone reading an old result knows which definition produced it. This is the single most important mechanism for comparability across a fast-moving roadmap, and it's why the warehouse-native tooling in section 2 matters.
• Guardrails that outlive any one test. A standing set of guardrails (latency, error rate, upgrade rate, privacy guardrail) applied to every experiment means the definition of "did we break something" doesn't get renegotiated per launch.
• Long-run holdouts. Keep a small, persistent holdout that doesn't get the stream of weekly changes, so you can measure the cumulative impact of rapid iteration — not just each change in isolation. It's the antidote to shipping ten individually-positive changes that sum to a regression.

✓ Check yourself

Answer from memory before opening each one.

1. Product wants to A/B test a new onboarding flow, and you hold no durable user ID. Name two randomization units and say what you measure.

   Show answer

   Use an anonymous session or device bucket, or a hash of the billing account ID server-side for logged-in surfaces; cluster or geo rollout works when even a bucket is too much identity. Measure aggregates per arm, such as day-2 return rate, because the unit of analysis is the arm and not the person. See privacy-safe A/B design.

2. Why is hashing a billing account ID acceptable for assignment while attaching a long-lived tracking ID to stitch outcomes is not?

   Show answer

   The billing ID is a first-party, consented relationship the company already holds. A stitching ID rebuilds the identity graph the product promises not to keep, and it works only by de-anonymizing. If a design needs that, change the design. See the line you must not cross.

3. Run the sample-size script with a 40% baseline and a 2-point minimum effect. About what should it print per arm, and what do you do if traffic supplies a third of that in a sane runtime?

   Show answer

   About 9,500 sessions per arm at 5% alpha and 80% power. If traffic cannot deliver it, say the test cannot be powered: raise the minimum effect, apply CUPED to cut variance, or choose a bigger swing. Do not run it underpowered and read the noise. See stats you must defend.

4. A teammate checks the dashboard daily and ships on day four when p drops below 0.05. Why is the result unreliable, and what are two fixes?

   Show answer

   Repeated looks inflate false positives, and four days misses the weekly usage cycle. Fix it with a fixed horizon and a stopping rule set in advance, or a sequential method built for monitoring. The sibling guide on Experimentation & A/B Testing goes deeper. See the design discipline.

5. With a very large sample, a 0.1% relative lift reaches p below 0.05. Do you ship it?

   Show answer

   Not on that alone. Statistical significance says the effect is unlikely to be noise, not that it is worth shipping. Pre-declare the effect worth shipping, report the effect size with a confidence interval, and weigh the cost of the change. See stats you must defend.

6. Ten weekly changes each shipped after a positive readout, yet conversion has drifted down over the quarter. What would have caught it?

   Show answer

   A small long-run holdout that never receives the stream of changes, which measures cumulative impact instead of each change alone. Versioned metric definitions and standing guardrails keep the readouts comparable across weeks. See robust to rapid iteration.

Takeaway

• You do not need identity to experiment; you need a randomization unit and an aggregate outcome, and both can be privacy-safe.
• Assign by anonymous session or device bucket, by a server-side hash of a consented billing ID, or by cluster, and never add a durable ID to stitch outcomes.
• Specify the experiment before the statistics: hypothesis, one primary metric, guardrails including a privacy guardrail, unit, minimum effect, runtime and stopping rule.
• Compute sample size from the minimum effect before launch; if traffic cannot power it, raise the effect or use CUPED instead of running a weak test.
• Report effect size with a confidence interval, because with enough traffic anything is significant.
• Versioned metrics, standing guardrails and a long-run holdout let rigorous experiments coexist with weekly shipping.

The one place the company can see individual-grain data is the public token economy, which is the next chapter. → On-Chain Analytics

Further reading

• Trustworthy Online Controlled Experiments — Kohavi, Tang and Xu, Cambridge University Press, 2020. The reference on design, pitfalls and metrics for online experiments.
• Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data — Deng, Xu, Kohavi and Walker, WSDM 2013. The original CUPED paper.
• NormalIndPower — statsmodels documentation, accessed 2026. The power calculation used in the chapter's sample-size script.
• How Not To Run an A/B Test — Evan Miller, 2010. A short, clear account of why peeking inflates false positives.
• Sequential Testing — GrowthBook docs, accessed 2026. How always-valid methods let you monitor a running test.
• Experiments — PostHog docs, accessed 2026. Assignment, exposure and results in the tool already in the stack.