Section B · Foundations

The Statistics You Actually Need

Not a statistics course — the specific ideas that decide whether your A/B result is real: why two groups differ even when nothing changed, and the machinery that tells signal from noise.

Everything varies — that is the whole problem

Split your users into two random groups and change nothing for either. Measure conversion. The two numbers will not be equal. They are never equal, because each group is a sample, and samples bounce around the true rate by chance. Toss a fair coin 100 times twice; you will not get 50/50 both times.

So when you run a real test and treatment converts at 10.3% versus control's 10.0%, the entire question is: is that 0.3 points a real effect, or the same chance bounce you would have seen with no change at all? Statistics is the set of tools for answering exactly that, and nothing more. Every term below is in service of distinguishing signal from the noise that is always present.

Two things govern how much a sample bounces: how variable the underlying behavior is (the variance), and how many observations you have (the sample size). More variance, more bounce; more data, less bounce. Hold those two levers in mind — they reappear in every formula.

Those two levers are enough because of one result, the Central Limit Theorem: whatever the shape of the raw behavior — clicks, revenue, session counts, almost always lopsided — the average over a group settles into a bell curve, and its spread is the variance divided by the square root of the sample size. That is why the normal-based tests below work on metrics that look nothing like a bell curve, and why the formulas carry only those same two terms. The theorem leans on a large sample and tails that are not too wild, so it frays on small tests and heavy-tailed metrics like revenue per user — the case the delta method in the next chapter handles.

The null hypothesis: assume you did nothing

Classical testing works by contradiction. You start from the null hypothesis: there is no difference between treatment and control — your change did nothing. Then you ask: if the null were true, how surprising is the difference I actually observed? If it would be very surprising — a bounce this big almost never happens by chance — you reject the null and conclude the change did something. If it would not be surprising, you fail to reject; the data is consistent with "no effect."

The asymmetry matters. You never "prove the null true." A quiet result means you did not find enough evidence of an effect, which is different from proving there is none — maybe the effect is small and you lacked the data to see it. This is why "not significant" must never be reported as "no difference." It is "no detected difference," and the gap between those is where a lot of bad decisions live.

The p-value, honestly

The p-value is the probability of seeing a difference at least as extreme as the one you observed, if the null hypothesis were true. Small p-value: your data would be unlikely under "no effect," so "no effect" looks wrong. By convention many teams reject the null when p < 0.05.

Now the part that trips up almost everyone. The p-value is not:

The probability the null is true. p = 0.03 does not mean "3% chance there is no effect." It is a statement about the data given the null, not about the null given the data. Flipping that conditional is the single most common statistical error in product reviews.
The probability your result is a fluke. Related flip, same mistake.
A measure of effect size. A tiny, useless effect can have a microscopic p-value if the sample is huge. p says "is there probably an effect," never "is it big enough to care."

The 0.05 line is a convention, not a law of nature — it sets how often you will cry wolf (see the error types below). Treat it as a decision threshold you chose, not a truth detector.

Confidence intervals: prefer them to bare p-values

A p-value gives a yes/no. A confidence interval gives a range of plausible effect sizes, which is almost always more useful. "Treatment lifted conversion by +0.3% (95% CI: +0.1% to +0.5%)" tells you the direction, the likely magnitude, and the precision, all at once.

The honest definition is slightly awkward: a 95% CI is an interval produced by a procedure that, run many times, would contain the true effect 95% of the time. Loosely and usefully: it is the range of effects the data cannot rule out. Two practical links to significance:

If the 95% CI excludes zero, the result is significant at p < 0.05. If it includes zero, it is not.
The width of the interval is your precision. A CI of +0.1% to +0.5% is a confident small win; a CI of −2% to +8% is a shrug dressed as a number, and usually means you need more data.

Report the interval, not just the point estimate and a star. A reviewer who sees "−2% to +8%" immediately knows the test was underpowered; one who sees "+5% ✱" does not.

The two kinds of error

Any decision rule that turns noisy data into a yes/no will sometimes be wrong, in two distinct ways:

Type I error (false positive) — you declare a winner when the change actually did nothing. Its rate is α, the significance level you set. Choosing p < 0.05 means accepting a 5% false-positive rate on true nulls: ship 20 genuinely useless changes under this rule and about one will look like a winner by chance.
Type II error (false negative) — there is a real effect, but your test fails to reach significance and you miss it. Its rate is β.

They trade off. Demand stronger evidence (smaller α) and you cry wolf less often but also miss more real effects (larger β). Which error is worse depends on the decision: for an irreversible, expensive change you want a small α; for cheap, reversible tweaks you might tolerate more false positives to avoid missing wins. The point is to choose deliberately rather than inherit 0.05 without a thought.

Power and the minimum detectable effect

Power is 1 − β: the probability your test detects a real effect of a given size, when one truly exists. A test with 80% power (the usual target) will catch a true effect of that size four times out of five and miss it one time in five. Running an underpowered test is the quiet killer of experimentation programs: you get a stream of "not significant" results, conclude your ideas do not work, and never learn that the test simply could not see effects of the size you were shipping.

Power is driven by four things, and fixing any three determines the fourth:

Effect size — bigger true effects are easier to detect.
Variance — noisier metrics are harder; reducing variance is a free power boost (CUPED, Chapter 05).
Sample size — more units, more power. This is the lever you usually control.
Significance level α — a stricter threshold costs power.

Turn this around and you get the number you actually plan with: the minimum detectable effect (MDE) — the smallest true effect your test can reliably catch, given your traffic and a target power. If your MDE is 2% and you only care about 5% wins, good. If your MDE is 2% and the change might realistically move the metric 0.5%, the test is hopeless before it starts, and you should know that on day zero. Here is the calculation for a conversion-rate (proportion) metric:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.10          # current conversion rate
mde_rel  = 0.05          # want to detect a 5% relative lift (0.10 -> 0.105)
treat    = baseline * (1 + mde_rel)

effect = proportion_effectsize(treat, baseline)   # standardized effect size
n_per_group = NormalIndPower().solve_power(
    effect_size=effect, alpha=0.05, power=0.80, alternative="two-sided")

print(round(n_per_group))   # ~58,000 users PER group for a 5% relative lift

Notice how sensitive it is: halving the effect you want to detect roughly quadruples the sample you need. That non-linear cost is why small sites struggle to test small changes, and why variance reduction and good metric choice matter so much. Designing an Experiment builds this calculation into the planning step.

Statistical significance is not practical significance

With enough traffic, everything becomes statistically significant. A giant consumer site can detect a 0.01% change — real, measurable, and completely not worth the engineering, the added complexity, or the risk. Statistical significance answers "is the effect probably non-zero?" It does not answer "is the effect big enough to act on?"

So pair every test with a practical significance threshold decided in advance: the smallest effect that would actually change your decision. Ship only when the result is both statistically significant and past that line — and read the confidence interval against the line too. A CI entirely above your threshold is a clear ship; a CI that is significant but sits below the threshold is a real-but-trivial effect you can usually skip. Deciding this number before the test is also your best defense against moving the goalposts once you see the result.

✓ Check yourself

Your test returns p = 0.03. A stakeholder says "so there's a 97% chance the feature works." Correct them precisely.
Answer
The p-value is the probability of data this extreme if the feature did nothing (the null) — not the probability the feature works. It says nothing directly about how likely the null is to be true, and nothing about the effect's size. A 97% "chance it works" would require a prior and Bayesian reasoning, not a frequentist p-value.
Why is "not statistically significant" different from "no effect"?
Answer
Failing to reject the null means you did not find enough evidence of an effect — which also happens when a real effect exists but the test was underpowered. "No detected effect" is consistent with both "no effect" and "an effect too small for this test to see." Reporting it as "no difference" throws away that distinction.
Your result is significant with a 95% CI of +0.05% to +0.15%, and your practical threshold is a 1% lift. Ship it?
Answer
No. It is statistically significant (CI excludes zero) but the entire plausible range sits far below your 1% practical threshold — a real but trivial effect. Significance without practical significance is not a ship signal.
You want to detect a 2% relative lift instead of a 4% one, traffic unchanged. Roughly what happens to the sample size or run time you need?
Answer
Required sample size scales roughly with the inverse square of the effect, so halving the detectable effect (4% → 2%) multiplies the needed sample by about four — and therefore about four times the run time at the same traffic. Small effects are expensive to see.
Two levers reduce how much a sample estimate bounces around the truth. Name them, and say which one variance reduction attacks.
Answer
Lower variance and larger sample size both reduce the bounce. Variance-reduction methods such as CUPED attack the variance lever, buying the same precision from less data — effectively free power without more traffic.

Takeaway

Two groups differ by chance even when nothing changed, so every A/B result is a signal-versus-noise judgment. The null hypothesis frames it; the p-value and — better — the confidence interval quantify it; the two error types and power tell you what the test can and cannot see before you run it; and practical significance keeps you from shipping real-but-pointless effects. Compute your MDE first: a test that cannot detect the effect you care about is a waste of a quarter. Next, turn these ideas into a concrete plan — hypothesis, metrics, randomization unit, and sample size — in Designing an Experiment.