Bayesian Decision Rule (Expected Loss)

What It Is

Instead of asking "is this statistically significant?", the Bayesian rule answers the two questions a stakeholder actually has: how likely is B better, and how much do I lose if I'm wrong? It needs no fixed sample size to be valid, which is the escape hatch for small-N marketing.

With a non-informative prior, after S successes and F failures the posterior conversion rate is Beta(S+1, F+1). From the two posteriors:

  • Probability B beats A: Pr(p_B > p_A) (closed form via the Beta function, or Monte Carlo).
  • Expected loss of shipping B: E[ max(p_A − p_B, 0) ] — how likely you're wrong, weighted by how costly the mistake is.

The Stopping Rule: Threshold of Caring

Ship the variant when its expected loss falls below a small threshold ε that the operator sets — the largest regret you're willing to absorb. ε is a business parameter, not a universal constant: set it per channel from the cost of being wrong. The rule degrades gracefully — on thin data the posteriors are wide, expected loss stays above ε, and the rule simply says "not yet" rather than manufacturing confidence.

Worked Numbers (from the source)

A page at ~300 visitors/arm: control 12/300 (4.0%), variant 21/300 (7.0%). A frequentist test of the pre-registered 5%→6% target would need ~8,155/arm (54 weeks) — unrunnable. The Bayesian rule, on the actual data: Pr(B>A) = 0.944, expected loss of shipping B = 0.046% conversion vs. 3.03% expected loss of staying on A. With ε = 0.1%, expected loss < ε → ship B. Data that is "insignificant forever" under NHST yields a clear, low-regret decision here.

How It Applies to Marketing Factory

This is the factory's default decision rule because it answers the business question, needs no unreachable sample size, and never over-claims on sparse evidence. It is the natural call when the minimum-detectable-effect gate fails. It also frames softer judgments — e.g. whether a new angle has found message-market-fit — as explicit probabilities and losses rather than gut calls. Pair with sequential-testing when continuous monitoring is wanted and traffic is sufficient.

Referenced from: small-n-experiment-design