Start Here

The one idea

You change something — a button, a ranking model, a price — and a number moves. The hard question is never did the number move. It is did my change cause the move, or would the number have moved anyway?

You can never observe the answer directly, because you cannot both ship the change and not ship it to the same people at the same time. That missing half — what would have happened without the change — is called the counterfactual, and estimating it is the entire game.

An experiment estimates the counterfactual by building a group that is just like the people who got the change, except they did not get it. The trick that makes the two groups comparable is randomization: flip a coin for each user. With enough users, the two groups end up balanced on everything — age, country, device, how much they already spend, the weather where they live, and a thousand things you never measured. So any difference in outcomes after the change is attributable to the change itself. That is the whole logic of an A/B test, and everything else in this guide is detail on how to do it without fooling yourself.

What we mean by "A/B test"

The names overlap, so pin them down once:

• Randomized controlled trial (RCT) — the general scientific method: randomly assign subjects to a treatment or a control, then compare. Medicine has done this for decades.
• A/B test — an RCT run on a product. "A" is the control (what you have now), "B" is the treatment (your change). Split traffic at random, show each group its version, compare a metric.
• Online controlled experiment (OCE) — the industry term for A/B tests run live on a website or app, usually at large scale and often many at once. This guide is mostly about OCEs, because that is where most teams work.

Same core idea at three scales. When this guide says "experiment," it means a controlled comparison with randomization. When it says "A/B test," it means the everyday product version of that.

The words you'll meet

You do not need these memorized yet — each gets a full treatment later — but a map helps:

• Control / treatment — the unchanged group and the changed group. Also called A and B, or baseline and variant.
• Variant — one version in the test. An A/B test has two; an A/B/n test has more.
• Metric — the number you compare. The single most important one is the overall evaluation criterion (OEC).
• Randomization unit — what you flip the coin on: a user, a session, a device, a city. This choice shapes everything.
• Lift / effect size — how much the treatment moved the metric, e.g. "+3.2% conversion."
• Statistical significance — the result is unlikely to be noise. Measured with a p-value or a confidence interval.
• Power — the test's ability to detect a real effect of a given size, if one exists.
• Guardrail metric — a number you are not trying to improve but must not harm (latency, crashes, revenue).

Chapter 11 closes with a fuller glossary; by then these will be second nature.

How this guide is built

Twelve chapters in four parts, each building on the last:

• Section A · Orient — this chapter and Why Experiment at All: the causal logic and when a test is the right tool.
• Section B · Foundations — the statistics, designing a test, running it, and reading the results. This is the core; read it in order.
• Section C · Getting it right — threats to validity, designs beyond the simple A/B, and what to do when you cannot randomize. This is where experience lives.
• Section D · In the world — industry playbooks (consumer, marketplaces, ranking and SaaS, fintech, mobile, health) and building the practice.

The core (Section B) is enough to run a sound test. The judgment that separates a trustworthy program from a number-generator is in Sections C and D.

Takeaway

An experiment answers one question — did my change cause this? — by using randomization to build a comparable control group that stands in for the counterfactual. Hold that idea; every technique ahead is a way to protect it. Start with Why Experiment at All, which shows why this beats the alternatives and where it still falls short.