Section B · Foundations

Running It Well

A perfectly designed experiment can still be worthless if assignment leaks, logging drops users, or someone stops it the moment it looks good. Operations is where trust is kept or lost.

Assignment and exposure

Assignment should be deterministic and stable: hash a stable identifier (user ID, or a persistent cookie/device ID for logged-out traffic) together with the experiment name, and map the hash to a variant. The same user always lands in the same bucket for the life of the test, assignment is independent across experiments, and you never have to store a giant table of who-got-what.

The subtler decision is when to count a user. Imagine a change to the checkout page. If you analyze everyone assigned to the experiment, you dilute the effect with people who never reached checkout and could not possibly be affected — their noise swamps the signal. The fix is triggered (exposure-based) analysis: count a user only from the point they actually hit the changed surface, and log that exposure event. Done right, this sharpens power and gives the honest effect on the affected population. Done carelessly, it is a bias risk — if "triggering" happens differently in treatment and control (say, the new code logs exposure a moment earlier), you have broken comparability. Log exposure at the same decision point in both arms.

Sample ratio mismatch: the first thing to check

You split 50/50. You end with 500,000 in control and 503,200 in treatment. Close enough? No. With numbers that large, a gap that size almost never happens by chance — a chi-square test against the expected 50/50 returns a vanishingly small p-value. This is a sample ratio mismatch (SRM), and it is the most important alarm in experimentation: if the ratio is off, something is broken, and the whole result is suspect.

SRM matters because whatever corrupted the ratio probably also corrupted the comparison. Common causes:

Broken assignment — a bug sends certain users (one browser, one country) disproportionately to one arm.
Differential loss — the treatment is slower or crashes more, so some treatment users drop out before being logged. The survivors are now a biased sample — often the more patient, higher-intent users — which can manufacture a fake win.
Redirect bias — treatment users are redirected to a new page and some bounce during the redirect; control is not, so it keeps everyone.
Bot or logging artifacts — bots hash into one bucket, or one arm's events are dropped by a pipeline.

Make an SRM check automatic and blocking: any experiment that fails it should refuse to report a result until the cause is found. A real SRM is not a nuisance to override — it is the test telling you its own answer cannot be trusted.

The A/A test: trust, but verify the system

Before you trust your platform, run an A/A test: two identical variants, no change at all. Two things should happen. The metrics should come out statistically indistinguishable — and across many A/A tests, you should see a significant result about 5% of the time at α = 0.05, no more (that is the false-positive rate working as designed). And the split should pass the SRM check.

A/A tests catch the embarrassing problems before they cost you a decision: a miscalibrated variance estimator that cries significance too often, a broken randomizer, correlated assignment across experiments, or a logging pipeline that loses events unevenly. If your A/A tests are significant far more than 5% of the time, your p-values are lying, and no real experiment on that platform can be trusted until you find out why.

Ramp-up and guardrails

Do not launch a risky change to 50% of users on day one. Ramp: expose 1%, watch the guardrails (errors, latency, crashes, revenue) for a few hours, then 5%, 20%, 50%. Early ramp stages are about safety and bugs, not statistics — you are catching the change that doubles page-load time or breaks on Safari before it reaches everyone. A kill switch that reverts to control instantly, without a deploy, is non-negotiable; the whole point of a controlled rollout is that you can stop it.

One subtlety: the stats clock effectively starts at your final allocation. Metrics measured at 1% exposure are noisy and often not representative (early adopters, caching effects), so plan your power at the allocation you will actually hold, and do not read the ramp stages as the experiment.

The peeking problem

Here is the trap that quietly ruins more results than any other. You planned a two-week test. On day 3 you peek; not significant. Day 5; not significant. Day 8; significant! — you ship. Feels responsible. It is statistically broken.

The 5% false-positive rate is guaranteed only if you look once, at the end. Every additional peek is another chance for the ever-wandering difference to cross the significance line by luck, and you have resolved to stop the instant it does. Peek daily for two weeks and your real false-positive rate climbs from 5% toward 20–30%. You are not detecting effects; you are waiting for noise to flatter you, then freezing the frame. This is why the fixed horizon set during design is sacred: decide the end in advance, then look once.

Tests you are allowed to peek at

But early stopping is genuinely valuable — to ship a big win sooner, or to kill a harmful change fast. The answer is not to peek at a fixed-horizon test; it is to use a test designed for continuous monitoring, which pays for the right to look by demanding stronger evidence:

Group sequential methods plan a fixed number of interim analyses and spend your total α across them (alpha spending, e.g. O'Brien–Fleming), so the overall false-positive rate stays at 5% no matter which look triggers the stop.
Always-valid / anytime-valid inference (sequential probability ratio tests, mixture SPRT, confidence sequences) gives p-values and intervals that are valid at every moment, so you can monitor continuously and stop whenever you like. This is what modern platforms such as those built on Optimizely's Stats Engine and others use under the hood.

The trade is honest: sequential tests need somewhat more data to reach the same conclusion as a fixed-horizon test that ran to completion, in exchange for the freedom to stop early and safely. The rule is simple — pick your monitoring policy before launch. Either commit to a fixed horizon and look once, or use a sequential method and peek all you want. What you may never do is run a fixed-horizon test and treat it like a sequential one.

✓ Check yourself

Why analyze from the point of exposure rather than everyone assigned to the experiment, and what is the risk of doing it badly?
Answer
Users who never reach the changed surface cannot be affected; including them dilutes the effect with pure noise and costs power. The risk is differential triggering: if treatment and control log exposure at different points, the two populations stop being comparable and you introduce bias. Trigger at the identical decision point in both arms.
A test shows a 2% lift and is significant, but control has 48,000 users and treatment has 52,000. What do you do before celebrating?
Answer
Stop — that is a sample ratio mismatch. A 48/52 split at this scale is not chance; something is broken (assignment bug, differential dropout, redirect bias, logging loss), and whatever caused it likely biased the 2% lift too. Investigate and fix the cause; do not trust the result until the ratio is right.
Across 200 A/A tests on your platform, 34 come back significant at α = 0.05. Is the platform healthy?
Answer
No. You would expect about 5% — roughly 10 — to be significant by chance; 34 (17%) means the system produces too many false positives, likely a miscalibrated variance estimate (e.g. treating a ratio metric naively) or correlated assignment. The p-values can't be trusted until it's fixed.
Explain, mechanically, why peeking daily and stopping on the first significant day inflates the false-positive rate.
Answer
The observed difference wanders randomly over time. The 5% guarantee assumes one look; each extra look is another independent chance for that wandering difference to cross the threshold by luck. A rule that stops the moment it crosses captures those lucky crossings, so the cumulative probability of a false positive grows far above 5%.
Your team genuinely needs to stop early to catch harmful changes fast. How do you get that without inflating error?
Answer
Use a method built for continuous monitoring — group sequential with alpha spending, or always-valid/anytime-valid inference — chosen before launch. These keep the overall false-positive rate controlled across unlimited looks, at the cost of needing somewhat more data for the same conclusion. The key is committing to the monitoring policy up front.

Takeaway

Execution is where trust is earned: deterministic stable assignment, exposure-based analysis logged identically in both arms, an automatic blocking SRM check, A/A tests to prove the platform's p-values are calibrated, a safety ramp with a kill switch, and — above all — a monitoring policy fixed before launch so you never peek your way into a false win. Get these right and the number your test returns is one you can defend. Now that you have a clean result, Reading the Results covers how to interpret it without the segmentation and multiple-comparison traps.