Section B · Foundations

Designing an Experiment

Most bad experiments were lost before they launched — a fuzzy hypothesis, a gameable metric, the wrong randomization unit, or too little traffic to ever conclude. Design fixes all four on paper.

A hypothesis worth testing

"Let's try a green button" is not a hypothesis; it is a change with no claim attached. A real hypothesis names the change, the mechanism you expect, and the metric that mechanism should move: "Making the primary call-to-action higher-contrast will make it easier to find, so more visitors will start checkout — raising checkout-start rate." Now the test can confirm or refute something specific.

Writing the mechanism down pays off twice. First, it forces you to predict which metric should move and roughly how much, which is the input to your sample-size math. Second, when the result comes back, you can tell a true win from a lucky one: if checkout-starts rose but for a reason unrelated to contrast (say, the change also removed a distracting banner), your mechanism was wrong even though the number went up, and you have learned something more durable than "green won." Vague hypotheses produce vague learnings that do not transfer to the next decision.

The OEC and the metric hierarchy

You will be tempted to watch twenty metrics and declare victory on whichever moved. That is how you ship noise (see multiple comparisons, Chapter 05). Decide the decision rule first, as a small hierarchy:

The OEC — overall evaluation criterion. The one metric (or a small weighted combination) that decides the test. A good OEC is sensitive (moves when the product gets better), aligned with long-term value (not just a short-term proxy you can juice), and hard to game. "Sessions per user this week" is sensitive but gameable — adding annoying interstitials can raise it while hurting the business. "Revenue per user" is aligned but noisy. Choosing the OEC is the most consequential design decision you make, and it is a judgment call, not a formula.
Secondary and diagnostic metrics. The numbers that explain why the OEC moved — funnel steps, feature usage, latency. They do not decide the test; they help you understand and trust it.
Guardrail metrics. Things you are not trying to improve but must not harm: page-load time, crash rate, error rate, revenue, unsubscribes, support tickets. A win on the OEC that trips a guardrail is usually not a win. Guardrails are also your early-warning system — a sudden guardrail breach often means a bug, not a real effect.

The discipline is to write these down before launch and commit to the OEC as the arbiter. A team that picks its winning metric after seeing the data is not experimenting; it is storytelling with numbers.

The shape of a metric matters

Two metrics that sound alike can behave very differently under analysis:

Averages of per-user values (revenue per user, sessions per user) are the friendly case: each user contributes one number, the randomization unit and the analysis unit match, and the standard two-sample test applies.
Ratio metrics (click-through rate as clicks ÷ impressions, where users contribute many impressions) are the trap. The denominator varies per user, so the simple formula understates the variance and your p-values come out too small — you will declare wins that are not there. These need the delta method or bootstrapping to get the variance right. You do not need the derivation, but you must recognize the shape: if the thing you are averaging is itself a ratio of two user-level counts, the naive test is wrong.
Heavy-tailed metrics (revenue, where one whale dwarfs thousands of users) have huge variance and are easily swung by a single outlier landing in one group. Capping (winsorizing) extreme values or analyzing a transformed metric is standard, and it also helps power.

Match your statistical test to the metric's shape. Most experimentation platforms handle this for you — but knowing it is why you cannot just paste CTR into a t-test and trust the star.

The randomization unit: what you flip the coin on

You can randomize by user, by session, by device, by page view, or by a cluster such as a city or an account. The choice has three consequences:

Consistency of experience. Randomize by page view and the same person might see the old design, then the new, then the old again within a visit — confusing, and it muddies the effect. Randomize by user (usually via a stable hashed ID) and each person has one coherent experience for the whole test. User-level is the default for anything a person would notice.
The analysis must match the unit. If you randomize by user, you analyze per user. Mixing — randomizing by user but testing as if each click were independent — reintroduces the ratio-metric variance error above and inflates significance. Unit of randomization = unit of analysis.
Coarser units cost power but can be necessary. Randomizing by city instead of by user gives you far fewer independent units (hundreds of cities, not millions of users), so much less power. You accept that cost when finer randomization would break the experiment — most often because of interference, where one user's treatment spills onto another (marketplaces, social features). Chapters 07 and 09 cover cluster and switchback designs built for exactly this.

Decide the unit from the change: Does one user's experience affect another's? Does the user need consistency across sessions? Answer those and the unit is usually obvious.

Sample size and duration, decided up front

From the previous chapter you have the machinery: baseline rate, the minimum effect worth detecting (MDE), the metric's variance, your α and target power give you the users-per-group you need. Compute it before launch. This is the single most protective habit in experimentation, because it tells you whether the test is even possible: if you need 58,000 users per group and you get 2,000 a week, this test takes a year, and you should either pick a bigger change, a more sensitive metric, or a different method.

Sample size sets the "how many"; the calendar sets the "how long," and three rules keep duration honest:

Run full weeks. Behavior differs by day — weekends, paydays, business-hours B2B usage. Stopping mid-week over-weights whatever days you happened to catch. Always run in whole-week multiples.
Cover at least one full business cycle, typically a week, often two. A test that ends before a weekly cycle completes measures a biased slice of your users.
Fix the horizon in advance, then wait. Decide the end date (or the sample target) before launch and do not stop early because it "looks significant." Peeking at a running test and stopping on the first green reading massively inflates your false-positive rate — the subject of the next chapter.

Traffic allocation is the last knob. A 50/50 split maximizes power for a two-arm test. Smaller treatment exposure (say 5%) limits risk for a scary change but needs much longer to reach the same power — a deliberate trade, not a free lunch.

✓ Check yourself

Turn "let's test a shorter signup form" into a proper hypothesis.
Answer
Something like: "Removing the optional phone-number field reduces friction, so more visitors who start signup will finish it — raising signup-completion rate — without hurting downstream activation." It names the change, the mechanism (less friction), the primary metric (completion rate), and a guardrail (activation).
Why is "weekly active sessions per user" a risky OEC, and what makes a metric hard to game?
Answer
You can inflate sessions with interruptions, notifications, or interstitials that fragment usage without creating value — the metric rises while the experience worsens. A hard-to-game OEC is tied to genuine user value or revenue and cannot be moved by degrading the product; that usually means it is noisier, which is the trade you accept.
You randomize by user but compute click-through rate as total clicks ÷ total impressions and run a plain two-sample test. What goes wrong?
Answer
The analysis unit (impressions) no longer matches the randomization unit (users), and impressions from the same user are correlated. The naive variance is too small, so p-values are understated and you will call wins that are noise. Use the delta method or bootstrap, or analyze a per-user metric.
When would you randomize by city instead of by user, despite the big loss of power?
Answer
When one user's treatment affects another's outcome — interference — so user-level randomization would contaminate the control. Marketplaces (treatment buyers consume supply control buyers wanted), social features, or anything with shared resources push you to cluster-level units even though cities are far fewer independent units than users.
Your sample-size math says you need 6 weeks to reach 80% power. Marketing wants to call it after 4 days because treatment is up and significant. What do you say?
Answer
Hold the pre-registered horizon. Stopping early on the first significant reading is peeking, which inflates the false-positive rate well beyond the nominal 5%; a 4-day "win" on a 6-week test is very likely noise, and the duration also hasn't covered a full weekly cycle. If early stopping is genuinely needed, use a sequential test designed to allow it (Chapter 04).

Takeaway

Design is where experiments are won: a hypothesis with a mechanism, an OEC that is sensitive and hard to game plus guardrails you refuse to harm, a statistical test matched to the metric's shape, a randomization unit chosen from whether users interfere and need consistency, and a sample size and whole-week duration computed before launch. Commit to all of it in writing, because every one of these decisions is also a place you could rationalize a better-looking result after the fact. With the design fixed, the risk moves to execution — assignment bugs, sample ratio mismatch, and the temptation to peek — which is Running It Well.