Section D · In the world

Playbooks I — Consumer, Marketplaces, Ranking

The principles are universal; the application is not. Three settings that bend the textbook in different directions — one by scale, one by interference, one by the gap between a click and real value.

Why the industry changes the test

The machinery from Sections B and C does not change across industries, but which parts bite hardest does. What varies is the traffic you have, the OEC that captures value, whether users interfere, how long effects take to appear, and what a "winner" even means. The next two chapters walk ten settings; this one takes the three that most shaped the discipline.

Consumer web and e-commerce

This is the native habitat of A/B testing — the world Amazon, Booking.com, and Microsoft's Bing built their experimentation platforms for — and it is the friendly case: millions of independent users, short funnels, and outcomes (purchase, signup, subscribe) that arrive within a session. High traffic means good power even for small effects, which is exactly the point: at scale, a 0.5% conversion lift is real money, and the whole program is built to detect and bank many small wins rather than chase rare big ones.

The playbook:

  • OEC. Revenue or conversion per user, not clicks. The cautionary tale is optimizing click-through on a product tile and discovering you drove clicks that did not convert. Tie the OEC to the transaction.
  • Guardrails. Latency above all — the classic finding is that added page-load time measurably reduces revenue, so a feature that wins on conversion but slows the page can be a net loss. Watch revenue, add-to-cart, and bounce as guardrails even when they are not the OEC.
  • Watch the shape of the metric. Much of e-commerce is ratio and heavy-tailed metrics — revenue is dominated by a few big spenders, click-through is clicks over impressions — so variance control (capping, CUPED) and correct variance estimation matter more than anywhere.
  • Novelty and seasonality. Promotions, holidays, and sales distort everything; run across representative periods and separate a seasonal spike from a real effect.

Done well, consumer experimentation is less about any single test and more about throughput: a platform that lets hundreds of well-powered tests run safely in parallel, each banking a small, trustworthy gain.

Two-sided marketplaces

Ride-hailing, food delivery, home-sharing, labor and freelance platforms, dating — anywhere supply and demand meet — break the one assumption that makes A/B tests simple: that users are independent. Treat one side better and you consume a shared, finite resource that the control side wanted. A naive user-level test therefore measures the wrong thing, usually overstating the launch effect, because part of the "gain" is value transferred from control rather than created.

The playbook:

  • Expect interference; design for it. Reach for switchback designs (flip the whole system over time windows) or geo/cluster randomization (whole cities in treatment or control) so the marketplace response is captured inside the unit. Accept the power hit — windows and cities are far fewer than users.
  • Measure both sides and the balance. A change that delights riders can crush drivers, or vice versa. Track demand-side and supply-side OECs plus a balance metric (fulfillment rate, match rate, wait time); a one-sided win is often a two-sided wash.
  • Beware cannibalization. A promotion that lifts one segment may simply pull volume from another. Marketplace effects are frequently zero-sum in the short run, so the question is always net, system-level impact.
  • Think long-term. Supply and demand re-equilibrate over weeks (drivers enter, prices adjust), so short tests can mislead. Geo holdouts over longer windows are common.

The through-line: in a marketplace the unit of experimentation is rarely the user, and pretending otherwise is the single most common and most expensive mistake.

Ads, search, and recommendations

Ranking systems — search results, feed, recommendations, ad selection — have their own distinct playbook because the thing you are testing is usually a model, the traffic cost of a bad ranker is high, and the easy metric (clicks) is a treacherous proxy for the real goal (satisfaction, long-term retention, advertiser value).

The playbook:

  • Screen with interleaving, decide with A/B. Interleaving compares two rankers within the same user's list and is one to two orders of magnitude more sensitive, so it is ideal for quickly narrowing many candidate models. The finalists still get a full A/B test on engagement and business metrics, because winning on blended clicks is not the same as making users better off.
  • Use offline replay to pre-filter, carefully. Teams score candidate rankers against logged data before touching live traffic, but logs were collected under the old ranker, so naive replay is biased toward it. Off-policy / counterfactual evaluation (inverse-propensity methods) corrects for this, and even then it is a filter, not a verdict — the online test is the ground truth.
  • Fight position bias. Higher-ranked items get clicked because they are higher, not necessarily better. Click metrics and interleaving constructions both have to account for position, or you will reward rankers for shuffling, not improving.
  • Pick metrics beyond the click. Optimize raw clicks and you breed clickbait. Serious ranking teams use satisfaction-oriented metrics (long dwell, task completion, return visits) and long-term holdbacks to guard against short-term engagement that erodes trust.
  • Mind auction interference in ads. Ad arms can share advertiser budgets and compete in the same auction, so one arm's behavior changes the other's — another reason naive user splits mislead, and why budget-split or geo designs appear here too.

✓ Check yourself

  1. An e-commerce test lifts product-tile click-through by 8% but revenue per user is flat. Ship it?
    Answer

    No. Clicks are a proxy; the OEC is revenue (or conversion) per user, and it did not move — the change likely drove clicks that did not convert. Optimizing the click metric here would be chasing a number disconnected from value. Flat revenue with more clicks may even signal wasted attention.

  2. Why does a user-randomized A/B test overstate the effect of a driver-incentive change in a ride-hailing app?
    Answer

    Interference over shared supply: treated riders/drivers consume matches and availability that control users would otherwise have gotten, so part of treatment's "gain" is transferred from control rather than created. At full launch there's no control to take from, so the real effect is smaller. Switchback or geo designs capture the true system effect.

  3. Why screen rankers with interleaving but still finish with an A/B test?
    Answer

    Interleaving is far more sensitive, so it cheaply narrows many ranker candidates by comparing them within each user's list. But it answers only "which gets more clicks on a blended list," which isn't the same as real user satisfaction or business value — so finalists get a full A/B test on engagement and business metrics before launch.

  4. What goes wrong if a recommendations team optimizes purely for click-through, and what guards against it?
    Answer

    It breeds clickbait — content engineered to be clicked, not to satisfy — which can erode trust and long-term retention. Guards: satisfaction-oriented metrics (dwell, task completion, return visits), position-bias correction, and long-term holdbacks to confirm short-term clicks don't come at the cost of durable value.

  5. In three words each, name the dominant challenge in consumer e-commerce, marketplaces, and ranking.
    Answer

    Consumer: variance and scale (small effects, ratio/heavy-tailed metrics, latency guardrails). Marketplaces: interference between units (shared supply, zero-sum, two-sided). Ranking: clicks versus value (proxy metrics, position bias, off-policy evaluation).

Takeaway

Same statistics, three different pressure points: consumer e-commerce is about banking many small, well-powered wins while controlling variance and guarding latency; marketplaces are about interference, which forces switchback or geo units and two-sided metrics; ranking is about the gap between a click and real value, met with interleaving, off-policy evaluation, position-bias care, and long-term holdbacks. Next, five more settings with their own quirks — subscription SaaS, fintech, mobile and games, lifecycle marketing, and clinical trials — in Playbooks II.

Further reading