Section A · Orient

Why Experiment at All

Every team already has data. So why go to the trouble of a controlled test? Because almost every other way of reading that data quietly answers a different question than the one you are asking.

Correlation is not causation

In summer, ice-cream sales and drownings rise together. No one thinks ice cream drowns people; a third thing — hot weather — drives both. That third thing is a confounder, and confounders are everywhere in product data, usually better disguised.

Take a claim you will hear in every product review: "users who use our collaboration feature retain 40% better, so we should push everyone toward it." Maybe. Or maybe the kind of team that adopts a collaboration feature is already more committed — bigger, better funded, further along in onboarding — and would have retained better regardless. The feature and the retention share a cause (an engaged team), exactly like the ice cream and the drownings. Push the feature on everyone and you may move nothing, because you were reading a correlation and acting as if it were causation.

This is not a rare trap. It is the default state of observational product data: the people who do X are different from the people who do not, in ways that also affect the outcome. The differences you can see (you can match on them). The ones you cannot see are the ones that ruin you — this is called selection bias, and no amount of data volume fixes it. A billion biased rows are still biased.

The counterfactual, and why it is impossible to observe

The clean way to state what you actually want: for the users who got your change, what would have happened to those same users, at the same time, if they had not? That is the counterfactual. The difference between what did happen and that counterfactual is the true causal effect of your change.

The catch — sometimes called the fundamental problem of causal inference — is that you only ever see one of the two worlds for any given user. You shipped the change to Ada, so you see Ada-with-change; you will never see Ada-without-change over the same week. The data for the counterfactual does not exist and never will.

So every method that claims a causal effect is really a method for estimating a counterfactual you cannot observe. They differ only in how believable that estimate is. An A/B test produces the most believable one available to a product team, and it does so with a surprisingly simple device.

Why randomization works

You cannot show Ada both worlds. But if you split a large population at random into two groups, the groups become statistically interchangeable: same mix of ages, countries, devices, tenure, spend, intent — and, crucially, the same mix of things you never measured and never will. The control group is not Ada's counterfactual, but in aggregate it is a faithful stand-in for what the treatment group would have done without the change.

The word "random" is doing heavy lifting. Matching on the confounders you can see — comparing treatment users to control users of the same age and country — only balances the known confounders. Randomization balances the unknown ones too, for free, because the coin flip knows nothing about the user. That is the property no observational method can buy at any price, and it is why a randomized test sits at the top of the evidence hierarchy.

Two conditions keep that property intact, and most of this guide is about protecting them. First, the assignment really must be random and respected — if treatment users systematically differ from control (a bug, a redirect, a crash that drops users), the balance breaks; running it well covers how this fails in practice. Second, one group's experience must not leak into the other's — if treatment users change the world that control users live in (shared supply, social features, a shared model), the control stops being a clean counterfactual; threats to validity deals with these interference effects.

The weaker alternatives, and what they cost

Experiments are work. Here is what teams reach for instead, and the specific way each one misleads:

Before-and-after (pre/post). Ship to everyone, compare last week to this week. Cheap and intuitive, and wrong whenever anything else changed with time — a holiday, a marketing push, a competitor outage, normal seasonality. You have confounded your change with the calendar, and you usually cannot tell by how much.
Cohort or segment comparison. Compare users who adopted the feature to those who did not. This is the collaboration-feature trap above: selection bias, dressed as analysis.
Observational modeling (regression, matching, propensity scores). Statistically "control for" the confounders. Genuinely useful, and the honest version of causal inference when you cannot randomize — but it can only adjust for confounders you measured and modeled correctly. The unmeasured ones remain, and you rarely know how large they are.
Expert judgment and the HiPPO. The highest-paid person's opinion. Fast and sometimes right, but the track record of even strong product intuition against controlled tests is humbling: across large programs, a large share of ideas that everyone expected to win are flat or negative when measured. Judgment is best spent on what to test, not on declaring the winner.

None of these are useless; they are how you generate hypotheses and how you decide in the many cases where a clean test is impossible. The point is to know which question each one answers, and to reach for randomization when the decision is worth the truth.

When an A/B test is the wrong tool

Experimentation is powerful, not universal. Reach for something else when:

You cannot randomize cleanly. A nationwide price change, a brand campaign, a one-time launch — there is no held-out control. Quasi-experiments (geo tests, difference-in-differences, synthetic control) are the honest fallback.
You do not have the units. Significance comes from sample size. A tool used by 200 enterprise accounts, or a change to a page 50 people a week see, may never reach a conclusion in a reasonable time. Chapter 03 shows how to compute this before you waste a quarter.
The effect takes too long to appear. Retention and lifetime-value effects can take months. You can still test, but you measure a short-term proxy and accept the risk that it diverges from the long-term truth — a tension long-term holdbacks address.
Interference is strong. In a marketplace or social network, helping treatment users can come straight out of control users' pockets. A naive A/B test overstates the effect; you need switchback or cluster designs (Chapters 07 and 09).
It is a one-way door, or an ethical line. Some decisions are too consequential, too irreversible, or too sensitive to split-test on real people. Knowing where that line sits is part of the craft (Chapter 11).

Most of the time, though, the decision is reversible, the traffic is there, and the honest answer is worth having. In those cases there is no substitute for a controlled experiment — and the rest of this guide is about running one you can trust.

✓ Check yourself

A teammate shows that users who enable notifications retain far better and wants to turn notifications on for everyone. What is the specific flaw in that reasoning, and what would change your mind?
Answer
Selection bias: people who choose to enable notifications are already more engaged, and that engagement — not the notifications — may drive the retention. The comparison confounds the treatment with the type of user. The fix is a randomized test that turns notifications on for a random half and compares retention; only then is the difference attributable to notifications.
Why does randomization protect you against confounders you never even measured, when matching and regression cannot?
Answer
The coin flip is independent of every user attribute, measured or not, so in expectation both groups get the same distribution of all variables. Matching and regression can only balance the confounders you put into the model; anything you did not measure stays unbalanced and biases the estimate.
State, in one sentence, what the control group is standing in for.
Answer
The counterfactual: what the treatment group would have done over the same period if it had not received the change. The control is not any individual's counterfactual, but in aggregate it estimates the treatment group's.
Give one situation where a before/after comparison and a randomized test would disagree, and say which to trust.
Answer
Ship a checkout redesign the same week a holiday sale starts. Before/after shows a big lift that is mostly the sale; the A/B test, which splits traffic within the same week, isolates the redesign's effect. Trust the test — the before/after has confounded the change with the calendar.
Name two conditions under which a randomized A/B test stops giving a trustworthy answer.
Answer
(1) Assignment is not clean — a bug, redirect, or crash makes treatment and control users systematically different. (2) Interference — treatment users change the environment control users experience (shared supply, social or network effects, a shared model), so the control is no longer a clean counterfactual.

Takeaway

Almost every way of reading product data answers "what is associated with the outcome," when the decision needs "what causes it." Randomization is the one device that estimates the unobservable counterfactual while balancing even the confounders you never measured — which is why a controlled test, when you can run one, beats cohorts, regressions, and intuition. Keep its two load-bearing assumptions in view: clean assignment and no interference. Next, build the statistical machinery that turns a difference between two groups into a defensible claim, in The Statistics You Actually Need.