Playbooks II — SaaS, Fintech, Mobile, Health

B2B SaaS: the low-traffic problem

Enterprise software inverts the consumer setup. You might have 500 accounts, not 50 million users, and the outcomes that matter — renewal, expansion, seat growth — resolve over quarters, not sessions. The power math is brutal: with a few hundred units you simply cannot detect small effects, and the business decisions are large and infrequent.

The playbook:

• Randomize at the right level, usually the account. Users within a company interact and share workflows, so user-level splits leak; the account (or workspace) is typically the clean unit — which makes your sample even smaller.
• Test what you can at the user level, decide the rest other ways. In-product UX changes (onboarding, a feature's design) can sometimes run as user- or account-level A/B tests on the larger self-serve segment. Big, rare outcomes (did this move renewals?) usually fall back to quasi-experiments, phased rollouts with difference-in-differences, and frankly qualitative evidence.
• Lean on proxies and leading indicators, validated against the long-term outcome where possible — activation, feature adoption, usage depth — because you cannot wait a year for every decision.
• Feature flags and staged rollouts are the operational backbone even when a clean test is impossible: ship to a cohort, watch guardrails, expand.

The honest summary: B2B teams experiment where traffic allows and triangulate elsewhere. Pretending 40 enterprise accounts can power a 2% test is the local version of the underpowered-test trap.

Fintech and pricing: rare events, real stakes, regulation

Financial products add three constraints at once. Outcomes are often rare and delayed (fraud, loan default, chargebacks happen to a small fraction of users, sometimes months later); the stakes are asymmetric (a false "approve" can cost far more than a false "decline"); and regulation and fairness limit what you may randomize at all — you generally cannot offer different prices or credit terms to otherwise-identical customers just to see what happens.

The playbook:

• Rare outcomes need huge samples or long windows. Detecting a change in a 0.5% fraud rate takes enormous traffic; plan for it, and lean on variance reduction and sometimes rare-event-aware methods.
• Guardrails are risk and compliance metrics, not just latency — approval rates, loss rates, complaint and dispute rates, fairness across protected groups. A conversion win that quietly raises default is a loss.
• Measure short-term proxies, verify long-term. You cannot wait for every loan to mature, so you model early-risk indicators and confirm with holdbacks and back-testing as outcomes realize.
• Respect the legal line. Where you cannot randomize pricing or terms, use geo tests, policy-change difference-in-differences, or carefully governed limited rollouts — and involve compliance in the design, not after.

Mobile and games: shipping friction and LiveOps

Native mobile apps add a constraint web does not have: you cannot instantly change the app a user is running, and app-store review sits between you and release. So experimentation depends on server-driven config / feature flags — ship the variants inside one build and switch them server-side — and on staged rollouts through the stores. You also contend with heavy device and OS fragmentation, so segmentation by platform and version is routine, and old app versions linger in the wild for months.

The playbook, with games as the extreme case:

• Retention is the OEC, usually as D1 / D7 / D30 retention and engagement, because mobile value compounds through continued use. These are delayed metrics, so tests run longer and novelty effects (a shiny new mechanic spikes then fades) are pronounced.
• Monetization is brutally heavy-tailed. In free-to-play games and apps, a tiny fraction of "whales" drive most revenue, so revenue metrics have enormous variance and a single big spender in one arm can swing a test. Cap, segment payers from non-payers, and treat revenue effects with extra skepticism.
• LiveOps is continuous experimentation. Live games run constant tests on events, offers, difficulty, and economy tuning — fast iteration where bandits and rapid A/B cycles fit, balanced against long-term retention and spend guardrails so you do not optimize this week's revenue at the cost of churn.

Email and lifecycle marketing: the holdout is the test

Outbound messaging — email, push, SMS, in-app campaigns — has a uniquely clean experimental move and a uniquely tempting mistake. The clean move: a send/no-send holdout. Randomly withhold the message from a slice of the audience and compare their downstream behavior to those who received it; the difference is the campaign's true incremental effect. The tempting mistake: judging a campaign by open or click rates, which measure engagement with the message, not whether it caused any additional purchases, logins, or retention you would not have gotten anyway.

The playbook:

• Always keep a holdout for anything you want to claim credit for — per-campaign for big sends, and a global "no marketing" holdout to measure the whole program's incrementality (the lifecycle version of the global holdout).
• Measure the downstream outcome, not the open. Opens and clicks are diagnostics; incremental conversions and retention are the OEC.
• Watch long-term and list-health guardrails. More sends can lift this campaign while raising unsubscribes and fatigue that quietly shrink the reachable audience — a classic short-term win, long-term loss. Frequency and send-time are themselves testable.

Healthcare and clinical trials: where this all came from

The randomized controlled trial predates online experiments by decades, and clinical trials run the same logic under far stricter discipline, because the cost of a wrong answer is measured in lives. It is worth studying precisely because the rigor is dialed to maximum, and several of its habits are ones product teams underuse.

What clinical trials do that product testing often skips:

• Pre-registration. The hypothesis, primary endpoint, sample size, and analysis plan are registered publicly before the trial, which forecloses the metric-shopping and p-hacking that quietly corrupt product analysis.
• Blinding. Patients and often clinicians do not know who got the treatment, to prevent expectations from shaping outcomes — the rigorous answer to placebo and novelty effects.
• A single pre-specified primary endpoint and strict multiplicity control, rather than a dashboard of forty metrics to pick a winner from.
• Pre-planned interim analyses with formal stopping rules (data safety monitoring boards, alpha spending) — exactly the sequential discipline that lets you stop early for overwhelming benefit or harm without inflating error.
• Intention-to-treat analysis. Analyze people by the group they were randomized into, even if they did not comply, because dropping non-compliers reintroduces selection bias — the same attrition trap, handled by rule.
• Ethics as a gate. Informed consent, institutional review, and equipoise (you may only randomize when there is genuine uncertainty about which arm is better). The product analogue — when is it acceptable to experiment on users, and with what consent — is Chapter 11.

Product teams rightly move faster and tolerate more risk than drug trials; the point is not to copy the bureaucracy but to steal the habits that cost little and buy a lot of trust — pre-registration, one primary endpoint, principled stopping, and intention-to-treat.

✓ Check yourself

1. Why can a B2B SaaS team rarely A/B test the effect of a change on renewals, and what do they do instead?
   Answer

   Too few units (hundreds of accounts) and outcomes that resolve over quarters make a renewal A/B test wildly underpowered, and users within an account interfere so the unit is the account — smaller still. Instead they A/B test in-product changes on the larger self-serve segment, use validated leading indicators (activation, adoption), and fall back to phased rollouts with difference-in-differences and qualitative evidence for the big, rare outcomes.

2. In free-to-play games, why are revenue metrics so hard to test on, and what do you do about it?
   Answer

   Revenue is extremely heavy-tailed — a few whales produce most of it — so variance is huge and one big spender landing in one arm can swing the result. Mitigations: cap/winsorize extreme values, segment payers from non-payers, prefer retention as the OEC, and treat revenue effects with extra skepticism and longer runs.

3. A lifecycle team celebrates a 45% email open rate. What did they fail to measure?
   Answer

   Incrementality. Opens measure engagement with the message, not whether it caused additional conversions or retention beyond what those users would have done anyway. The fix is a send/no-send holdout and measuring the downstream outcome difference — plus watching unsubscribe/fatigue guardrails.

4. Name two clinical-trial habits product teams underuse, and the product failure each prevents.
   Answer

   Pre-registration (fixing hypothesis, primary endpoint, and analysis before the test) prevents metric-shopping and p-hacking. Intention-to-treat (analyzing by assigned group regardless of compliance) prevents the selection bias from dropping non-compliers. Others: a single primary endpoint (multiplicity), and pre-planned stopping rules (peeking).

5. What does "measure a short-term proxy, verify long-term" look like in fintech lending, and why is it unavoidable?
   Answer

   Loans take months to default, so you can't wait for every outcome before deciding. You model early-risk indicators as a proxy, make the call, and then confirm with holdbacks and back-testing as real defaults materialize — adjusting if the proxy diverged. It's unavoidable because the true outcome is both rare and delayed, while decisions must be made now.

Takeaway

Each setting bends the method around its binding constraint: SaaS around too few units (account-level tests, proxies, quasi-experiments); fintech around rare, delayed, regulated outcomes (risk guardrails, long-horizon verification, legal limits on randomization); mobile and games around shipping friction and heavy-tailed monetization (server config, staged rollouts, retention OEC); lifecycle marketing around incrementality (the send/no-send holdout); and clinical trials around the cost of being wrong (pre-registration, blinding, one endpoint, principled stopping, intention-to-treat). The habits worth stealing everywhere come from the strictest field. Finally, how to turn all of this into a program that an organization can trust — the platform, the culture, and the ethics — in Building the Practice.

Further reading

• Ron Kohavi, Diane Tang, Ya Xu — Trustworthy Online Controlled Experiments (2020), Chapters 5 (low-traffic, proxies) and 21 (triggering, heavy tails).
• U.S. National Library of Medicine — ClinicalTrials.gov, the public registry that makes trial pre-registration enforceable.
• Suzanne Senn — Statistical Issues in Drug Development (2007). Blinding, endpoints, interim analysis, and intention-to-treat, from the source discipline.