Section C · Getting it right

Threats to Validity

Everything in Section B can be done correctly and the experiment can still mislead you. These are the failure modes that survive a clean setup — the ones experience teaches and textbooks underweight.

Twyman's law: if it's amazing, it's probably wrong

"Any figure that looks interesting or different is usually wrong." A button-colour change that lifts revenue 30%, a copy tweak that doubles signups — the base rate of such effects in mature products is near zero, so the base rate of such readings being a bug is high. Before you celebrate a shocking result, assume instrumentation error and try to disprove it: check for SRM, look for a logging change that double-counts treatment, confirm the metric definition did not shift, see whether the effect is concentrated in one implausible segment. Most spectacular wins die in this step. The ones that survive scrutiny are the real discoveries — and now you can trust them.

Novelty and primacy effects

Two opposite time-based distortions, both caused by the change being new rather than better:

Novelty effect. A shiny new feature or redesign draws clicks simply because it is unfamiliar. Engagement spikes, then decays toward baseline as the novelty wears off. Ship on the early spike and you have overpaid for a fad.
Primacy effect (change aversion). The opposite: loyal users have muscle memory for the old flow and are annoyed or slowed by the change at first. The early metric dips, then recovers (or exceeds baseline) as they adapt. Kill on the early dip and you have thrown away a real long-term win.

Both show up as an effect that drifts over time, so the defenses are the same: plot the treatment effect by day rather than reading a single pooled number; segment new versus existing users (novelty and primacy mostly hit existing users, since new users have no "old way" to compare); and when the early period looks unrepresentative, run longer, or analyze only users who joined after launch to get the steady-state effect. If the daily effect is still trending when you stop, you do not yet know the real effect.

Interference: when one user's treatment leaks onto another

The whole logic of an A/B test assumes a user's outcome depends only on their own assignment — formally, the stable unit treatment value assumption (SUTVA). Plenty of products violate it:

Marketplaces (two-sided). Give treatment buyers a better ranking and they snap up the limited supply; control buyers now face a depleted marketplace and look worse than they would have. The measured "lift" is partly just supply moved from control to treatment — a zero-sum illusion that vanishes when you launch to everyone.
Social and communication features. A treatment user who posts more generates content that control users (their friends) see and react to, dragging control up. The gap between arms shrinks, hiding a real effect.
Shared systems. A treatment that hammers a shared cache, database, or ML model can slow or change behavior for control users too.
Budgets and pacing. In ads, treatment and control can draw from the same advertiser budget; one arm spending more starves the other.

The symptom is that the naive A/B over- or under-states the true launch effect, and you cannot tell by how much from the experiment alone. The fixes change the unit: randomize by cluster (city, region, social community) so interference stays inside a unit, or use switchback designs that alternate the whole system between treatment and control over time. Recognizing interference is a design-time job — ask "can one user's experience affect another's outcome?" before you launch, not after.

Seasonality and whether the effect generalizes

An experiment is internally valid when it correctly measures the effect for the users and period it ran on. External validity is whether that effect holds elsewhere — next quarter, for other users, at full scale. They are different, and a clean test guarantees only the first.

Common external-validity threats: a test run entirely during the winter holidays measures holiday shoppers, not your typical users; a test run on 5% of traffic may not reproduce at 100% if the change strains a shared system (a primacy of scale problem); a result from one country or platform may not transfer to another. None of these means the test was wrong — it means the question "will this effect persist and generalize?" needs its own evidence: run across a representative period, confirm big wins with a follow-up or a long-term holdback (Chapter 07), and be explicit about the population you measured.

Winner's curse and regression to the mean

Among experiments that just barely cross the significance line, the reported effect sizes are systematically too big. The reason is selection: a borderline result gets over the bar partly because random noise happened to push it up this time. Re-run it, and the noise is as likely to push down — so the effect "regresses to the mean" and the replication comes back smaller, sometimes to nothing. This is the winner's curse.

Practical consequences: do not take the point estimate from a marginal win to the bank for your revenue forecast — it is probably inflated. The protections are a priori power (well-powered tests have less inflation), skepticism toward effects that clear the bar by a hair, and replication or a holdback for anything you are betting the roadmap on. The same logic explains why the best result in a big batch of experiments is usually overstated: you selected it for being extreme.

Differential attrition and survivorship

If users drop out of the two arms at different rates — or different kinds of users drop out — the groups you end up comparing are no longer the groups you randomized, and comparability is gone. A treatment that is slightly slower may quietly shed impatient users; the ones who remain are more patient and higher-intent, so treatment looks better for a reason that has nothing to do with the feature. This often shows up first as an SRM, which is exactly why SRM is the alarm you never override.

Survivorship is the same bias in a wider lens: analyzing only the users, sessions, or accounts that "made it" to some point, when making it is itself affected by the treatment. The guard is to anchor analysis at the randomization point and the exposure point, track attrition by arm as a guardrail, and be suspicious whenever the thing you are conditioning on could have been changed by the treatment.

✓ Check yourself

Your redesign shows a +18% revenue lift on day one. Walk through your first five minutes.
Answer
Invoke Twyman's law: an 18% revenue jump is implausible, so assume a bug. Check for SRM, a changed or double-counted revenue metric, whether the effect is concentrated in one odd segment, and whether a logging change inflated treatment. Only after it survives that scrutiny — and ideally a few more days to rule out novelty — do you treat it as real.
A new feature's daily effect starts at +10% and declines toward +1% over two weeks. What is likely happening, and should you ship on the +10%?
Answer
A novelty effect — the feature drew attention for being new, and that is wearing off. Shipping on the +10% overestimates the durable value. Read the steady-state (the later days, or users who joined after launch); the honest effect is closer to +1%.
In a ride-hailing marketplace, you give treatment riders better pricing and measure a big lift in completed rides. Why might launching to everyone disappoint?
Answer
Interference: treatment riders consumed limited driver supply that control riders would otherwise have gotten, so part of the "lift" was supply transferred from control, not created. At 100% there is no control to take from; the zero-sum portion disappears. A switchback or geo (cluster) design measures the true launch effect.
Distinguish internal from external validity with a one-line example of a test that has the first but not the second.
Answer
Internal validity = the effect is correctly measured for the users and period tested; external = it generalizes. Example: a checkout change tested only during Black Friday is internally valid (it really helped holiday shoppers) but may lack external validity (it may do nothing for ordinary-week users).
Why is the effect from a barely-significant test probably overstated, and what should you do before forecasting revenue from it?
Answer
Winner's curse: marginal results cross the bar partly thanks to favorable noise, so the estimate is inflated and will regress toward the mean on replication. Before forecasting, discount it, prefer well-powered tests, and replicate or run a long-term holdback for anything you are betting on.

Takeaway

A statistically clean experiment can still deceive: suspect spectacular results (Twyman), separate "new" from "better" (novelty and primacy), ask before launch whether users interfere (SUTVA), keep internal and external validity distinct, discount marginal wins (winner's curse), and watch for differential attrition that quietly un-randomizes your groups. The common thread is that these are caught by design and interpretation, not by the p-value. Several of them — interference especially — force you past the simple two-arm test entirely, which is Beyond the Simple A/B.