Sequential & Always-Valid Testing

The Problem They Solve: Peeking

A standard A/B dashboard reports significance as if the sample size were fixed in advance. Stopping a test the moment it "looks significant" breaks that assumption and inflates false positives badly — Miller's simulation shows a fixed-horizon test stopped early hitting a 26.1% false-positive rate against a claimed 5%. An agent that monitors continuously is the maximum-inflation case, so it can never call a fixed-horizon test early.

SPRT (Sequential Probability Ratio Test)

Choose N up front, assign 50/50, track treatment successes T and control successes C:

  • Stop, declare treatment the winner when T − C ≥ 2√N (one-sided α=0.05; 2.25√N two-sided).
  • Stop, declare no winner when T + C ≥ N.

The threshold comes from d* ≈ z_{α/2}·√N. Because it stops as soon as the gap opens, it spends ~17.4% fewer observations on average than the fixed-sample test. Caveat: SPRT counts only successes, so it works best at low conversion rates and degrades at high ones.

Always-Valid Inference (mSPRT)

The mixture SPRT produces always-valid p-values and confidence intervals — numbers that remain valid at any stopping time, so you may read them whenever you like without inflating error. This is the method behind production engines like Optimizely's Stats Engine. For continuous monitoring it is the only frequentist tool that is safe to peek at by construction.

(Mechanism is textbook-established; specific production false-positive figures attributed to commercial engines are reported, not independently verified — see the source's provenance note.)

How It Applies to Marketing Factory

Agents watch dashboards in real time, which makes naive fixed-horizon testing actively dangerous. When a test passes the minimum-detectable-effect gate and will be monitored continuously, the factory uses always-valid inference; for low-rate conversions where early stopping saves real budget, SPRT. Choosing and respecting the stop rule is an agent-ownable, mechanical task; interpreting a borderline result is not (see agent-ownership-boundary). For business-framed ship/kill calls on thin data, pair with bayesian-decision-rule.

Referenced from: small-n-experiment-design