Pilot & Rollout

Phase 0 — Sandbox bake-off (pre-signature)

This happens before you sign the contract, during the negotiation phase. It's the single highest-yield activity in the whole evaluation. RFP responses are claims; the bake-off is evidence.

What you run

Take a fixed sample set of synthetic verifications — covering each jurisdiction in scope, each document type you expect, each demographic mix, plus deliberate failure cases — and run it through each shortlisted vendor's sandbox. Score the results yourself.

Sample set composition (200–500 cases total)

What to record per vendor

• Decision (approve / decline / review / indeterminate) for every case.
• Time-to-decision (median, p95).
• Vendor reason codes (do they map sensibly to your taxonomy?).
• Confidence score where available.
• API errors, timeouts, retries.
• Cost per check (compare against quoted pricing).

Engineer time

Allow 1 engineer week per vendor for sandbox setup + integration + running the sample set. Time-to-first-verification is itself a scored criterion — a vendor whose sandbox eats 3 days is signalling integration cost.

Phase 1 — Pilot design (post-signature, pre-full-launch)

The pilot is where you stop simulating and start running real production traffic. The goal: validate every assumption from the RFP and sandbox in your live environment, with real users, before you depend on the vendor for revenue.

Pilot structure

Each gate must pass exit criteria (below) before advancing. Plan ~9–10 weeks from go-live to 100%. Don't rush it; the only thing more expensive than a slow rollout is a fast one that breaks.

Sample sizes — how long does each stage need?

Sample size depends on what you're trying to detect. The numbers below assume p=0.05, 80% power, and the typical metric ranges for consumer IDV. Use them as a planning floor.

How to plan

• Estimate your daily verification volume.
• Decide which metrics are deal-breakers (typically: completion rate, false-reject rate).
• Compute the days at each % level needed to hit stat-sig on each metric.
• Pick the longer of: (sample-size duration) and (operational-load duration — i.e., long enough that support/dispute volume stabilizes).

What to measure

Build the dashboards in 04 § Observability before launch. These are the metrics that gate progression at each stage.

Primary metrics (deal-breakers)

Secondary metrics (watch list)

• Dispute rate — users contesting a decline.
• NPS / CSAT on the verification flow (in-product survey).
• Cost per successful verification (blended unit cost / completion rate).
• Sanctions hit rate and hit-investigation throughput.
• Webhook delivery success rate (≥ 99.9% required).
• API error rate by vendor 5xx / 4xx / timeout.
• Geographic distribution of decline rate — surface jurisdiction-specific regressions.

Tertiary metrics (post-launch monitoring)

• Downstream fraud rate on approved users (lagging indicator; needs 30–90 days).
• Re-verification trigger rate.
• Cohort-aged unit economics.
• Vendor invoice variance vs forecast.

Pilot exit criteria — gates between stages

Each gate is a hard yes/no. Fail any criterion at a gate, you stay at that % level until it's resolved. Don't soften the gates under pressure; that's how bad launches happen.

• Completion rate ≥ pre-pilot baseline – 2 pts
• False-reject rate ≤ pre-pilot baseline + 0.5 pt
• p95 time-to-decision ≤ 2.5 min for auto-decisions
• Webhook delivery ≥ 99.9% during measurement window
• Vendor API error rate ≤ 1% on 5-min windows, no sustained spikes
• Manual review queue did not exceed 4× baseline peak depth
• Support ticket volume on verification topics within 1.5× baseline
• Dispute rate within 1.5× baseline
• No vendor SLA breach during stage
• No P1 incidents attributable to the integration
• Compliance / MLRO sign-off on the audit trail review (random sample of 100 decisions)
• Cost per successful verification within 10% of forecast

Phased rollout: 0% → 1% → 10% → 50% → 100%

How to assign users to stages

Use a deterministic hash of user_id mod 10,000 — gives you fine-grained control. Sticky assignment is mandatory — a user who starts on the new vendor stays on it, even if their verification fails and they retry.

import hashlib

def route_vendor(user_id: str, rollout_pct: float, sticky: bool = True) -> str:
    """
    rollout_pct: 0.0–100.0, what % of users should go to new vendor.
    Sticky: same user always gets the same answer for the same rollout_pct.
    """
    h = int(hashlib.sha256(f"vendor-rollout:{user_id}".encode()).hexdigest()[:8], 16)
    bucket = (h % 10000) / 100.0   # 0.00–99.99
    return "vendor_new" if bucket < rollout_pct else "vendor_old"

# Wired to LaunchDarkly / Statsig / your config service:
ROLLOUT_PCT = config.get("idv.new_vendor.pct", default=0.0)

Stage-by-stage actions

Geo-staging playbook for multi-jurisdiction launches

If you operate in 5+ countries, don't ramp everywhere simultaneously. Stage by jurisdiction. Order matters.

Ordering principles

1. Start where you have low regulatory exposure. A country where you're not yet fully licensed, or where IDV is voluntary rather than mandatory, is safest first.
2. Start where the vendor is strongest. Vendor performance varies enormously by country; sequence to early wins.
3. Match volume to operations capacity. Don't ramp your largest country on day 1; you'll overwhelm manual review.
4. End with your most strategic country. By the time you're rolling out in your top market, you've absorbed all the lessons.

Example sequence (illustrative, US-headquartered fintech expanding to EU + LATAM)

Rollback criteria

Rollback is not failure — it's the system working. Pre-commit to rollback triggers so the team isn't deciding under pressure. Anyone on the on-call rotation should be able to invoke rollback unilaterally based on these triggers.

Automatic rollback triggers

• Vendor API error rate > 5% sustained 10 minutes.
• p95 latency > 5× normal sustained 15 minutes.
• Completion rate down > 5 points day-over-day on a stable population.
• Webhook delivery failure rate > 1% sustained 30 minutes.
• Vendor declares P1 incident.

Human-judgment rollback triggers

• Manual review queue backing up beyond ops capacity.
• Cluster of user reports indicating a regression invisible to metrics.
• Compliance / MLRO observes audit-trail issues.
• Regulator inquiry.
• Unit cost spiking beyond 20% of forecast.

Rollback procedure

1. Page on-call + IDV PM + MLRO/CCO via the same channel (don't fragment).
2. Confirm trigger from dashboards (5-minute timebox, then act).
3. Flip idv.new_vendor.pct to 0 in your config service.
4. Verify in monitoring that new traffic is on old vendor within 5 minutes.
5. For in-flight verifications: let them complete naturally; do not re-route mid-flow.
6. Comms: internal post in #incidents within 15 minutes. External comms only if user-facing impact > 5 minutes.
7. Post-incident: RCA within 5 business days. Decide go/no-go on next attempt.