Section C · Getting it right

Beyond the Simple A/B

The two-arm, fixed-split test is the workhorse, not the whole stable. When you have many variants, interference, ranking problems, or long-horizon questions, a different design is the right tool.

A/B/n and multivariate tests

An A/B/n test is the obvious extension: one control and several treatments (C, D, E…) at once. It is efficient when you have genuinely distinct candidates and the traffic to power each against control — but remember each extra arm is another comparison, so correct for multiplicity and keep the control large enough to compare against.

A multivariate test (MVT) is different: you vary several factors at once — headline (2 options) × image (2) × button (3) — and test combinations. Its payoff is detecting interactions: maybe the bold headline only works with the small image. A full factorial of those factors is 2×2×3 = 12 cells, and powering twelve cells needs a lot of traffic, which is MVT's main cost. Use MVT when you specifically suspect factors interact and you have the volume; otherwise a sequence of focused A/B tests is simpler, better powered per decision, and easier to reason about. "One factor at a time" misses interactions but is cheap; factorial finds them but is hungry — choose from whether interactions actually matter for your decision.

Multi-armed bandits: optimize while you learn

A fixed A/B test sends half your traffic to the loser for the entire run — fine when you want a precise effect estimate, wasteful when you just want the best option live as fast as possible. A multi-armed bandit instead adapts: it continuously shifts traffic toward whichever variant is performing best, while still exploring the others enough to keep learning. Algorithms like Thompson sampling and epsilon-greedy formalize this explore-versus-exploit balance.

Bandits shine for short-lived, many-option decisions where the cost of showing a loser is real: which of ten promotional banners to run this weekend, headline selection on a news site, ad creative rotation. They minimize regret — the cumulative cost of not having shown the best arm — rather than producing a clean, unbiased effect estimate. That last point is the catch: because assignment is adaptive and correlated with performance, bandits are built for making the decision, not for measuring exactly how much better the winner is, and naive significance tests on bandit data are biased. They also assume a fairly stationary world; if the best arm changes with time of day or audience, a plain bandit can chase noise. Rule of thumb: bandit when you want to optimize a disposable choice; fixed A/B when you need a trustworthy effect size to inform a lasting decision.

Switchback designs: testing the whole system over time

When interference is severe — a ride-hailing dispatch algorithm, surge pricing, a delivery assignment system — there is no clean way to split users, because any treated user changes the shared marketplace for everyone. The unit has to become time. A switchback design flips the entire system (or an entire region) between treatment and control in alternating time windows: treatment this half-hour, control the next, back and forth, ideally randomized. Each window is one observation, and comparing treatment windows to control windows captures the full system-level effect including the marketplace response.

The costs are real: your effective sample is the number of windows, not the number of rides, so power is limited; windows are correlated and need care in analysis; and you must leave buffer time for the system to re-equilibrate after each switch. But for marketplace and pricing problems it is often the only design that answers the actual question — "what happens to the whole system when we launch this?" — which is why Lyft, DoorDash, and similar companies lean on it. Combining switchback in time with geo (region) splits gives more units.

Interleaving: the right tool for ranking

Comparing two search or recommendation rankers with a standard A/B test is possible but blunt: whole-user assignment means a lot of between-user noise, so you need enormous traffic to tell a slightly better ranker from a slightly worse one. Interleaving is dramatically more sensitive. Instead of showing user A ranker X and user B ranker Y, you show each user a single result list blended from both rankers, then see which ranker's results earn the clicks. Every user is their own control, which removes the between-user variance that dominates ranking A/B tests.

The sensitivity gain is large — teams report needing one to two orders of magnitude less traffic to reach the same confidence as a classic A/B test, which is why search and feed teams use interleaving to screen ranking candidates quickly. The limits: it answers a narrow question (which ranker gets more clicks on the blended list), it needs careful construction to be fair (team-draft or balanced interleaving to avoid position bias), and it is a screening tool — the finalists usually still get a full A/B test on real engagement and business metrics before launch, because clicks on a blended list are not the same as long-term satisfaction.

Long-term holdbacks and holdouts

Most tests measure a week or two, but the decisions that matter — did this feature help retention, did a year of small wins actually compound — play out over months. Two designs reach for the long horizon:

Long-term holdback. After a change wins and ships to ~95% of users, keep a small randomized holdback group on the old experience for weeks or months. Comparing the two over time tells you whether the short-term win persisted, decayed (novelty), or grew (primacy recovery) — and gives you the durable effect, not the launch-week effect.
Global holdout. Hold a small slice of users out of all new features for a quarter, then compare them to everyone else. This measures the cumulative impact of the entire roadmap — the number leadership actually cares about — and catches the uncomfortable case where a year of individually "winning" tests added up to little, because each was measured on an optimistic short-term proxy.

Holdbacks are how a program keeps itself honest about the gap between short-term proxies and long-term value — the tension raised back in Chapter 01. They cost you a bit of exposure and some coordination, and they are worth it for anything strategic.

Choosing a design

A quick map from problem to tool:

Precise effect of one change, lasting decision → fixed A/B.
A few distinct candidates, enough traffic → A/B/n (correct for multiplicity).
Factors you suspect interact, lots of traffic → multivariate.
Optimize a short-lived, many-option choice fast → bandit.
Marketplace / pricing / strong interference → switchback or geo clusters.
Compare rankers sensitively → interleaving, then confirm with A/B.
Does the effect last / did the roadmap pay off → long-term holdback / global holdout.

The design follows the question. Reaching for a plain A/B when the problem has interference or a long horizon is the most common way a sophisticated team still gets the wrong answer.

✓ Check yourself

When is a multivariate test worth its traffic cost over a series of A/B tests?
Answer
When you specifically care whether factors interact — the combination matters, not just each factor alone — and you have enough traffic to power every cell of the factorial. If you only need each factor's main effect, sequential A/B tests are simpler and better powered per decision.
Why are bandits great for picking this weekend's promo banner but a poor choice for measuring how much a new checkout flow lifts revenue?
Answer
Bandits minimize regret by shifting traffic to the apparent winner, which is ideal for a disposable, short-lived choice. But that adaptive, performance-correlated assignment biases effect estimates, so they don't give the clean, trustworthy magnitude you need to justify a lasting decision like a checkout redesign — use a fixed A/B for that.
Your dispatch-algorithm A/B test (randomized by rider) shows a big win. Why might you switch to a switchback design, and what becomes your sample size?
Answer
Rider-level randomization suffers interference — treated riders change the shared marketplace for control riders, so the estimate is contaminated. A switchback flips the whole system between treatment and control over time windows to capture the true system effect. Your effective sample size becomes the number of time windows (often further multiplied by regions), not the number of rides.
Why can interleaving detect a better ranker with far less traffic than an A/B test?
Answer
It blends both rankers' results into one list per user and sees which ranker earns the clicks, making each user their own control. That removes the between-user variance that dominates ranking A/B tests, so far less traffic is needed — though it answers only a narrow clicks question and finalists still get a full A/B test.
A year of green A/B tests shipped, each a "win." How would you check whether they actually added up?
Answer
Use a global holdout: a small slice of users kept off all new features for the period, compared to everyone else. If the holdout is barely behind, the individually-significant wins didn't compound as advertised — often because each was measured on an optimistic short-term proxy rather than durable value.

Takeaway

Match the design to the problem: A/B/n and multivariate for multiple options and interactions, bandits to optimize disposable choices, switchback and geo clusters for interference, interleaving for sensitive ranking comparisons, and holdbacks for the long-horizon questions a two-week test cannot touch. Each buys something and costs something; naming the trade is the skill. All of these still rest on randomization — so the last gap to close is what to do when you genuinely cannot randomize at all, which is When You Can't Randomize.