Section B · Foundations

Reading the Results

The test finished and the dashboard says "+2.4%, significant." The real work starts now: what the number means, which slices to trust, and the three ways analysis manufactures wins that are not there.

Read the effect, not the star

Lead with the point estimate and its confidence interval, translated into the business: "checkout-start rose from 10.0% to 10.24%, +2.4% relative (95% CI: +1.1% to +3.7%), about +X orders a week at current traffic." That sentence carries direction, magnitude, precision, and consequence. The green "significant" badge carries almost none of it.

Two habits here. First, be explicit about relative versus absolute: a "+2.4%" that means 10.0%→10.24% (relative) is very different from one that means 10.0%→12.4% (absolute, +2.4 points). Teams ship the wrong thing by confusing these. Second, always check the result against your pre-set practical-significance threshold: significant-but-tiny is not a ship. The CI is your friend — read where it sits relative to that threshold, not just whether it clears zero.

Segmentation without fooling yourself

Effects are rarely uniform. A change can help new users and hurt power users; win on Android and lose on iOS; help in one country and do nothing in another. These heterogeneous treatment effects are real and worth finding — the overall average can hide a segment you are harming badly.

The danger is that segmentation is a multiple-comparison engine in disguise. Slice a flat result twenty ways and, by chance alone, roughly one slice will look "significant" at p < 0.05. Grab that slice, tell a story ("it works great for returning mobile users in Canada!"), and you have shipped noise with a narrative. Discipline:

Pre-register the segments you will examine and why, before seeing the data. A segment you hypothesized is a finding; a segment you went fishing for is a hypothesis at best.
Test for interaction, not just per-slice significance. The right question is "does the effect genuinely differ across segments?" — a statistical interaction test — not "is the effect significant within this one slice?"
Respect the power cost. Each segment has a fraction of the users, so each is underpowered. Small slices are noisy; treat a lone significant slice among many as a lead to re-test, not a conclusion.

Use segmentation to protect (catch a harmed group) and to generate hypotheses (a promising slice to confirm in a fresh, targeted test). Do not use it to rescue a flat experiment.

Simpson's paradox

A genuinely unsettling one: the treatment can win in every segment yet lose overall — or win overall while losing in every segment. It is not a glitch; it is what happens when the segments have different sizes and baseline rates and the mix of users differs between arms.

The classic cause in experiments is a ramp or an SRM. Suppose you ramped treatment over the test: early days have mostly one kind of user, later days another, and treatment and control ended up with different proportions of each because of when they were exposed. Aggregate the whole period and the mix difference can swamp or reverse the within-group effects. The practical rules: if your arms have a clean 50/50 split and no SRM, the aggregate is trustworthy and the paradox is unlikely. If the mix differs between arms, distrust the aggregate and look within homogeneous segments — and figure out why the mix differs, because that is usually a bug worth fixing.

Variance reduction: the free lunch (CUPED)

Recall that precision comes from lower variance or more data. You usually cannot get more users, but you can often cut the variance — and that is the same as buying more users for free. The standard technique is CUPED (Controlled-experiment Using Pre-Experiment Data).

The intuition: a user's behavior during the test is strongly predicted by their behavior before it. A user who spent a lot last month will probably spend a lot this month, change or no change. So subtract off the part of each user's outcome that their pre-experiment history already predicts, and analyze what remains — the part the experiment could actually have caused. Because you removed a big chunk of the natural, change-unrelated variation, the metric gets much quieter, the confidence interval tightens, and the same effect becomes detectable with far less data. In practice CUPED commonly cuts variance substantially, meaning materially shorter tests or the ability to detect smaller effects — and it is unbiased, because pre-experiment data cannot have been affected by a treatment that had not started. It is close to a free lunch, which is why most mature platforms apply it by default.

Multiple comparisons: the quiet inflator

Every extra thing you test against the same data is another lottery ticket for a false positive. The 5% rate is per comparison. It compounds across:

Many metrics — watch 20 metrics on a flat change and about one will look significant by chance.
Many variants — an A/B/C/D/E test is several comparisons against control at once.
Many segments — the slicing problem above.
Many experiments over time — a program running hundreds of tests will produce false winners steadily; this is why surprising single wins deserve replication.

Two defenses. First and most important: designate one primary metric (the OEC) as the decision-maker, set during design; everything else is diagnostic, not a licence to declare victory. Second, when you genuinely must make several formal comparisons, correct for them: Bonferroni (divide α by the number of tests) is simple and strict; controlling the false discovery rate (Benjamini–Hochberg) is less conservative and usually the better fit when you are screening many metrics and can tolerate a known fraction of false leads. The goal is not statistical purity for its own sake — it is to stop a dashboard of forty numbers from handing you a "win" every single time.

✓ Check yourself

Rewrite "treatment won, +5%, p = 0.01" into a sentence a product lead can act on.
Answer
Something like: "Treatment raised checkout-start from 10.0% to 10.5%, a +5% relative lift (95% CI: +2% to +8%), roughly +N orders/week — comfortably above our 1% practical threshold, so ship." It gives direction, magnitude, precision, business impact, and a decision against the threshold, which the bare p-value does not.
You slice a flat experiment into 20 segments and one is significant at p < 0.05 with a nice story. How much should you believe it?
Answer
Barely. With 20 independent slices you expect about one false positive at α = 0.05 even if the true effect is zero everywhere. Unless the segment was pre-registered and survives an interaction test, treat it as a hypothesis to confirm in a fresh, targeted experiment — not a conclusion.
Treatment beats control within new users and within returning users, but loses in aggregate. Name the phenomenon and the first thing you check.
Answer
Simpson's paradox. First check the arm composition — a sample ratio mismatch or a ramp that left treatment and control with different proportions of new versus returning users. If the mix differs, the aggregate is untrustworthy; fix the cause and trust the within-segment picture.
CUPED cut your metric's variance by 40%. In plain terms, what did you just gain?
Answer
Roughly the same precision you would have gotten from ~40% more users — so a materially shorter test, or the ability to detect a smaller effect in the same time. It works by removing the variation already predicted by each user's pre-experiment behavior, and it is unbiased because that history predates the treatment.
Your experiment dashboard reports 25 metrics. Why is "at least one is significant" nearly guaranteed even for a do-nothing change, and what is the primary fix?
Answer
Each metric is its own 5% false-positive lottery; across 25 roughly-independent metrics the chance that none fires is about 0.95²⁵ ≈ 28%, so a false positive somewhere is the common case. The primary fix is to pre-designate a single OEC as the decision-maker and treat the rest as diagnostic; when several formal comparisons are truly needed, apply a correction (Bonferroni or Benjamini–Hochberg).

Takeaway

Reading results well means leading with the effect and its interval against a practical threshold, segmenting to protect and to hypothesize rather than to rescue, recognizing Simpson's paradox as a mix problem to diagnose, using CUPED to buy precision for free, and refusing to let a wall of metrics and slices hand you a guaranteed "win." One instinct ties it together: when a result is surprising, suspect the measurement before the miracle. That instinct is the gateway to Threats to Validity, where we catalogue the ways a technically clean experiment still tells you something false.