Data Science · Deep dive

Experimentation & A/B Testing

From the counterfactual to trustworthy online experiments — the statistics, the design choices, the pitfalls that fool smart teams, the advanced designs for when a plain A/B will not do, and how the craft changes across ten industries.

Randomize, then compareall usersrandom splitA · controlB · treatment10.0%10.3%Δ = effectthe random split makes A and B comparable — even on what you never measured

An A/B test is the closest thing product teams have to a controlled lab experiment: split users at random, change one thing for one group, and measure the difference. Done well, it is the only tool that tells you whether your change caused the outcome rather than something else moving at the same time. Done carelessly, it produces confident numbers that are wrong — and wrong in ways that are hard to catch.

This guide takes you from zero to fluent. It assumes no statistics background, builds the ideas in order, and does not stop at the textbook: most of the hard-won knowledge is in the pitfalls (Section C) and in how real teams run experiments across consumer apps, marketplaces, search ranking, fintech, mobile games, and clinical trials (Section D).

Section A · Orient
00 Start Here The one mental model, how the guide is built, and three reading paths from beginner to practitioner. 01 Why Experiment at All Correlation versus causation, the counterfactual, and when an A/B test is the right tool — and when it is not.
Section B · Foundations
02 The Statistics You Actually Need Hypothesis testing, p-values, confidence intervals, the two error types, power, minimum detectable effect, and statistical versus practical significance. 03 Designing an Experiment A sharp hypothesis, the overall evaluation criterion and metric hierarchy, the randomization unit, and the sample-size calculation. 04 Running It Well Instrumentation, assignment, sample ratio mismatch, guardrail metrics, the peeking problem, and fixed-horizon versus sequential tests. 05 Reading the Results Estimating the effect, confidence intervals, honest segmentation, Simpson's paradox, variance reduction with CUPED, and multiple comparisons.
Section C · Getting it right
06 Threats to Validity Novelty and primacy, seasonality, network and interference effects, spillover, dilution, and Twyman's law — why a result too good to be true usually is. 07 Beyond the Simple A/B A/B/n and multivariate tests, multi-armed bandits, switchback designs, interleaving for ranking, and long-term holdbacks. 08 When You Can't Randomize Quasi-experiments: difference-in-differences, regression discontinuity, synthetic control, geo experiments, and matching.
Section D · In the world
09 Playbooks I — Consumer, Marketplaces, Ranking Consumer web and e-commerce, two-sided marketplaces, and ads, search, and recommendation ranking — three settings that break the textbook in different ways. 10 Playbooks II — SaaS, Fintech, Mobile, Health B2B SaaS, fintech and pricing, mobile and games, email and lifecycle marketing, and healthcare's randomized controlled trials. 11 Building the Practice The experimentation platform, a culture that trusts its own numbers, institutional memory, and the ethics and governance of testing on people.

How to read this guide

Chapters build on each other, so a first read in order pays off. If you are short on time, pick a path:

Complete beginner

Read 00 → 05 in order; that is the whole core. Then read 06 (the pitfalls) twice. Skim Section D for the industry closest to your work.

Practitioner brushing up

Skim 02, read 04 and 05 closely (SRM, peeking, CUPED, multiple comparisons), then 06, 07, 08. Section D is the payoff.

Interview in < 24 hours

00, 02, 03, 04, 05 for the mechanics; 06 for the traps interviewers love; one Section D chapter for your target industry.