Section C · Getting it right

When You Can't Randomize

A nationwide price change, a brand campaign, a one-time launch, a regulated feature — sometimes there is no held-out control. Quasi-experiments reconstruct the counterfactual from observational data, honestly, when a true A/B test is impossible.

When randomization is off the table

Plenty of real decisions cannot be split-tested. You can only change a nationwide price once, for everyone. A Super Bowl ad reaches whoever it reaches. A new regulation applies to all users on a date. A feature is too entangled to serve two versions. In each case there is no randomized control group — but the question is unchanged: what would have happened without the change?

Quasi-experimental methods estimate that counterfactual from data you can observe, by finding or constructing a comparison that stands in for the missing control. They are weaker than a randomized test — every one rests on an assumption you cannot fully verify — but done carefully they are far better than a naive before/after, and they are the honest tool when randomization is impossible. The craft is knowing which method's assumption is most defensible for your situation, and stating it out loud.

Difference-in-differences

The workhorse. You have a group that got the change (say, users in regions where you launched a feature) and a group that did not, and data from before and after. A plain before/after on the treated group confounds the change with everything else that moved in time; a plain treated-versus-untreated comparison confounds it with how the groups differ. Difference-in-differences (DiD) subtracts both biases: take the treated group's before→after change, subtract the untreated group's before→after change over the same period, and the control group's trend cancels out the time effects (seasonality, macro shifts) that would otherwise contaminate the result.

DiD lives or dies on the parallel-trends assumption: absent the change, the two groups would have moved in parallel. You cannot prove it, but you can support it by showing the groups tracked each other for a long stretch before the change. If their pre-period trends already diverged, DiD is not credible. It is the standard method for staggered geo rollouts, where the not-yet-launched regions serve as the control for the launched ones.

Geo tests and synthetic control

When the thing you are changing is inherently market-level — marketing spend, brand, pricing — the natural unit is a geo: a city, a media market, a region. A geo experiment randomizes or carefully selects which markets get the treatment and compares them to the rest, often analyzed with DiD. It is how marketing teams measure the incremental sales from a campaign that cannot be served to half of one city's residents but can be turned on in half the cities.

The elegant refinement is synthetic control. Instead of comparing your one treated region to a single messy control region, you build a synthetic control: a weighted blend of many untreated regions chosen so that, in the pre-period, the blend closely tracks the treated region's history. After the change, the gap between the real treated region and its synthetic twin is the estimated effect. It shines when you have one or a few treated units and a good pool of candidates to blend from — a single state rolling out a policy, one market getting a campaign — and it makes the counterfactual visible and auditable, which is part of why it has become a standard tool for marketing incrementality and policy evaluation.

Regression discontinuity

Sometimes a treatment is handed out by a hard threshold on some score: customers with a loyalty score above 80 get a perk; loans above a cutoff get manual review; students above a test line get a scholarship. A naive comparison of perk-getters to non-getters is hopeless — they differ by exactly the thing that earned the perk. But look right at the cutoff: a customer at 79.9 and one at 80.1 are, for all practical purposes, identical, yet one got the treatment and one did not. The threshold acts like a local coin flip.

Regression discontinuity (RD) compares outcomes just above versus just below the cutoff to estimate the effect for users near the threshold. Its assumption is clean and partly checkable: nothing else should jump discontinuously at the exact cutoff, and users must not be able to precisely manipulate their score to land on the favorable side (if they can, the people just above are different from those just below, and the local-randomization story breaks). RD's limit is scope — it tells you the effect near the cutoff, not for users far from it — but where a threshold rule exists, it is one of the most credible quasi-experiments available.

Matching and propensity scores

The most tempting and the weakest. When you have only observational data — some users adopted a feature, others did not — you can build a comparison group by matching each treated user to untreated users who look the same on observed characteristics, or by propensity-score methods that match on the modeled probability of having been treated. Done well, this balances the confounders you measured and is a real improvement over comparing raw averages.

But it can only ever balance what you observed and modeled. The reason a user adopted the feature — motivation, intent, context — is usually partly unmeasured, and those unmeasured drivers are exactly the ones that also move the outcome. This is the selection bias of Chapter 01, and no matching fixes it, because you cannot match on what you did not record. Use matching when nothing better is available, lean on rich covariates, and never present it with the confidence of a randomized test.

The honesty clause

Every method here replaces randomization's guarantee with an assumption: parallel trends (DiD), a good synthetic fit (synthetic control), no manipulation and no other jump at the cutoff (RD), no unmeasured confounding (matching). None is fully testable. So the professional practice is threefold: state the assumption explicitly so reviewers can judge it; test its implications where you can (pre-trends, placebo cutoffs, placebo periods, leaving out units); and triangulate — if DiD, a synthetic control, and a matched analysis all point the same way, believe it more; if they disagree, you have learned the result is fragile. And when a randomized test is feasible, prefer it — a mediocre experiment usually beats a sophisticated quasi-experiment, because randomization answers the one question these methods can only assume away.

✓ Check yourself

You launch a feature in 5 states and want its effect. Why is a before/after on those 5 states misleading, and how does difference-in-differences repair it?
Answer
Before/after confounds the launch with everything else that changed over time (seasonality, macro trends). DiD subtracts the before→after change in a comparison group (the not-yet-launched states) over the same window, so shared time effects cancel and what remains is the launch effect — valid as long as the two groups would have trended in parallel absent the change.
What single piece of evidence most supports a DiD analysis, and what finding would sink it?
Answer
Support: the treated and control groups tracked each other closely in the pre-period (parallel pre-trends). Sinks it: the groups' trends already diverged before the treatment, which means they would not have moved in parallel and the control is not a valid stand-in.
A perk is given to loyalty scores ≥ 80. Why compare users at 79–80 to 80–81 instead of all perk-holders to non-holders?
Answer
All perk-holders differ systematically from non-holders by the very score that earned the perk — a confounded comparison. Right at the cutoff, users just below and just above are essentially identical except for treatment, so the threshold acts as a local randomizer. That's regression discontinuity; the catch is users must not be able to precisely manipulate their score.
Why is propensity-score matching weaker than a randomized test even with a great model?
Answer
It balances only observed, modeled confounders. The reasons people self-select into treatment are usually partly unmeasured, and those unmeasured drivers also affect the outcome — residual selection bias that matching cannot remove, because you can't match on what you didn't record. Randomization balances unmeasured confounders too; matching cannot.
You cannot randomize a nationwide pricing change. What would a defensible, honest analysis look like?
Answer
Use a quasi-experiment — e.g. a geo test or synthetic control that builds a weighted blend of comparison markets tracking the treated market's pre-period, or DiD against markets not yet changed — state the assumption explicitly, run placebo checks (fake cutoffs/periods, leave-one-out), and triangulate across methods. Report it as an estimate under stated assumptions, not as a randomized result.

Takeaway

When you cannot randomize, you reconstruct the counterfactual: difference-in-differences nets out time via a comparison group, synthetic control builds a bespoke twin from many units, regression discontinuity exploits a threshold as a local coin flip, and matching balances observed confounders as a last resort. Each trades randomization's guarantee for a stated assumption — so name it, test what you can, triangulate, and reach for a real experiment whenever one is possible. With the full method toolkit in hand, the rest of the guide is about applying it: how experimentation actually plays out across industries, starting with consumer, marketplaces, and ranking.