Small-N Experiment Design — Deciding on Thin Data¶

For factory operators. A factory shipping 2–3 posts, emails, or page variants per week almost never accumulates the sample size a textbook A/B test demands. This document is the decision layer for that reality: how to call a winner (or kill a variant) validly on thin data, when to skip the test entirely, and how synthetic panels (synthetic-consumer-panels) buy signal you can't get from live traffic. It powers the v1 experiment loop and feeds the SSR calibration-loop.

The one-sentence thesis: at factory volumes, statistical significance is usually unreachable — so stop chasing it and switch to decision rules that stay valid at any sample size (sequential, always-valid, and Bayesian) plus an explicit "don't test this" rule.

1. The Peeking Trap (why the dashboard lies)¶

Every standard A/B dashboard reports significance as if the sample size were fixed in advance. The moment you watch a running test and stop when it "looks significant," that assumption breaks and the reported numbers become fiction.

Evan Miller's simulation is the canonical demonstration: a 50% baseline conversion test, stopped at first-significance or 150 observations, produces a 26.1% false-positive rate — more than 5× the claimed 5% (Miller, How Not To Run An A/B Test). To actually hold 5% under repeated looks you'd have to tighten the reported threshold drastically: ~2.9% for a single peek, ~1.0% for ten peeks.

Why the factory is the worst-case offender: an agent monitors continuously. Continuous peeking with a fixed-horizon test is the maximum-inflation scenario. The fix exists precisely for this: always-valid inference is built so results can be read at any stopping time without inflating error (Johari, Pekelis & Walsh, Always Valid Inference, arXiv:1512.04922). Production adopters of this approach report false-positive rates falling from over 20% to under 5% after the switch — a reported, secondary figure (see Provenance Note), but directionally consistent with Miller's simulation.

Rule 0: an agent may never call a fixed-horizon test early. Either fix N and wait, or use a method built for continuous monitoring (§4).

2. The Brutal Arithmetic of Small N¶

Before designing a test, compute whether it can ever finish. The two-proportion sample size (per arm), two-sided α=0.05, 80% power, is:

n_per_arm = (z_α/2 + z_β)² · [ p₁(1−p₁) + p₂(1−p₂) ] / (p₂ − p₁)²
          = (1.96 + 0.84)² · [ p₁(1−p₁) + p₂(1−p₂) ] / (p₂ − p₁)²

Computed for this KB (reproduce with the formula above):

Two facts fall out, and they define the whole strategy:

1. Effect size dominates everything. Required N scales with 1/(effect)². Halving the effect you want to detect quadruples the sample. Small wins are unmeasurable at low traffic; only big swings are detectable.
2. Channel traffic decides feasibility. A page at 150 visits/arm/week can't run even a generous CTA test inside a year. The same statistical question on an email list of thousands resolves in one send. Test where you have volume; decide-by-judgment where you don't.

Corollary — the Minimum Detectable Effect gate. Invert the formula: given your weekly traffic and a maximum run time you'll tolerate, solve for the smallest effect you could detect. If your plausible effect is smaller than that MDE, the test is undecidable — do not run it (see §7).

3. The Toolkit — Four Methods and When Each Applies¶

3a. Sequential testing (SPRT) — Miller's procedure¶

Choose N up front, assign 50/50, track treatment successes T and control successes C (Miller, Sequential A/B Testing):

• Stop, declare treatment the winner when T − C ≥ 2√N (one-sided α=0.05; use 2.25√N two-sided).
• Stop, declare no winner when T + C ≥ N.

The threshold derives from d* ≈ z_{α/2}·√N. Worked from Miller's own table: to detect a 10% lift at 80% power, N=2,922, d*=106 (note 2√2922 ≈ 108, the conservative approximation). Because it can stop as soon as the gap opens, it spends on average 17.4% fewer conversions than the fixed-sample test. Caveat: it only counts successes, so it shines at low conversion rates and degrades when rates are high.

3b. Always-valid inference (mSPRT) — check whenever you want¶

The mixture SPRT produces always-valid p-values and confidence intervals: numbers you can read at any point during the test without inflating error (Johari, Pekelis & Walsh, arXiv:1512.04922). It is the method behind Optimizely's Stats Engine. For an agent that monitors continuously, this is the only frequentist tool that is safe to peek at by construction. Use it when you want anytime-valid CIs and have a platform that computes them.

3c. Bayesian decision-making — the factory default¶

Bayesian inference needs no pre-ordained sample size to be valid, which is exactly the small-N escape hatch. With a non-informative prior, after S successes and F failures the posterior is Beta(S+1, F+1) (Miller, Bayesian A/B Testing). From the two posteriors you compute the two numbers a stakeholder actually asks for:

• Probability B beats A: Pr(p_B > p_A) (closed form via the Beta function, or Monte Carlo).
• Expected loss of shipping B: E[ max(p_A − p_B, 0) ] — how likely you're wrong × how much it costs when you are (the expected-loss decision rule, Stucchio, Bayesian A/B Testing at VWO).

Stopping rule = threshold of caring. Ship the variant when its expected loss falls below a small business threshold ε that the operator sets — the largest regret you're willing to absorb (e.g. ε = 0.1% absolute conversion). ε is a business parameter, not a universal constant: set it per channel from the cost of being wrong. This rule degrades gracefully: on thin data the posteriors are wide, expected loss stays above ε, and the rule simply says "not yet" — no false confidence.

3d. Multi-armed bandits — when you'd rather earn than learn¶

If the goal is to maximize cumulative outcome during the test (ongoing content/subject optimization) rather than cleanly attribute a winner, shift traffic toward the leader as evidence accrues (Thompson sampling over the Beta posteriors is the natural fit). You capture value while still learning, at the price of clean end-of-test inference. Use bandits for perpetual optimization loops; use Bayesian decision (3c) for one-off ship/kill calls.

4. Explicit Stopping Rules (the gate)¶

Every factory experiment declares its stop rule before launch. No rule → no test.

Three parameters are set per experiment, up front, and logged: MPE (minimum practical effect worth shipping), ε (threshold of caring), and max run time (the patience ceiling, from the MDE gate in §2).

5. Two Worked Examples¶

Example A — Landing-page CTA at low traffic (Bayesian beats the unreachable frequentist test)¶

A page gets ~300 visitors/arm over two weeks. Control A: 12 conversions / 300 (4.0%). Variant B: 21 / 300 (7.0%).

• Frequentist verdict: the test was pre-registered to detect a 5%→6% lift (its MPE), which needs 8,155/arm (§2) — 54 weeks at this page's traffic. The test is unrunnable; a fixed-horizon dashboard reads "not significant" forever, even though B is over-performing the target.
• Bayesian verdict (posteriors Beta(13, 289) and Beta(22, 280); 200k Monte-Carlo draws, computed for this KB): Pr(p_B > p_A) = 0.944. Expected loss of shipping B = 0.046% conversion; expected loss of staying on A = 3.03%. With a threshold of caring ε = 0.1%, expected loss (0.046%) < ε → ship B. The downside of being wrong is a twentieth of a percentage point; the downside of inaction is sixty times larger.

The lesson: the same data that is "insignificant" forever under NHST yields a clear, quantified, low-regret decision under an expected-loss rule.

Example B — Email subject line where the test is feasible¶

An email goes to 2,000 recipients/arm. Hypothesis: a curiosity-gap subject lifts open rate from 20% to 25% (+25% rel).

• Feasibility (§2): detecting +25% rel needs 1,091/arm — covered in a single send. Run it as a normal fixed-horizon test on the next campaign; the decision is clean.
• If the hypothesised lift were only +10% (20%→22%): now 6,507/arm, ~3.3 sends. Either batch three sends before deciding, or — because an agent will be tempted to peek between sends — run it as always-valid mSPRT (3b) so reading the result after each send stays valid. Stop rule: ship when the always-valid CI for the lift clears the +10% MPE; kill if it excludes it; abandon at the max-run-time of 4 sends.

Contrast with A: when traffic is sufficient relative to the effect, classic testing works — use it. The exotic machinery is for the case where it doesn't.

6. When NOT to A/B Test¶

Skipping a doomed test is a positive decision, not a failure. Do not run a live A/B test when:

• The MDE gate fails (§2): your plausible effect is below the smallest effect detectable within max run time. Running it just burns weeks to reach "inconclusive."
• The change is obviously right or cheap to reverse: ship it, keep a small holdout, monitor. Decision cost < test cost.
• You'd need to test more than one variable at once on thin traffic: low traffic cannot support multivariate noise. Test one visible behavior at a time, or not at all.

When you skip, you still decide — using a prior + practical judgment, a small holdout to monitor for harm, and ideally an SSR pre-screen (§7) to get some comparative signal. The governing principle throughout: practical significance over statistical significance. A reliably-measured 0.3% lift you can't ship profitably is worth less than a confident directional call on a 30% swing.

7. SSR Synthetic Panels as a Pre-Screen¶

When live N is unreachable, generate signal instead of waiting for it. Semantic-Similarity-Rating synthetic panels (synthetic-consumer-panels, semantic-similarity-rating, embedding-similarity) score variants against anchor-statements before any real traffic is spent — turning "we can't test 5 subject lines on 200 sends" into "rank 5 subject lines synthetically, then live-test only the top 2."

Two hard constraints carry over from the SSR ingest, and they are non-negotiable:

• Comparative use only. SSR predicts relative ranking (B > A), not absolute conversion rates. Use it to choose what to test live, never to replace the live decision.
• Mandatory calibration-loop. Every synthetic verdict that later gets a real outcome feeds back to measure synthetic-vs-live agreement; uncalibrated panels are subject to domain-transfer-risk and silently drift. No calibration record → SSR output is advisory only.

This is the bridge between thin live data and a decision: SSR narrows the field, the live experiment (§3–5) confirms the survivor, and the calibration loop keeps the synthetic stage honest.

8. The Factory Experiment Loop (machine-readable)¶

The pieces assemble into one repeatable loop the agent-workflow-pattern can execute:

experiment_loop:
  trigger: new_variant_proposed            # content/email/page variant enters queue
  steps:
    - id: feasibility_gate
      action: compute_MDE(weekly_traffic, max_run_time)
      branch:
        plausible_effect < MDE: -> ssr_decide   # undecidable live; go synthetic
        else:                    -> design_test
    - id: ssr_pre_screen                     # optional, recommended when >2 variants
      action: rank_variants(anchor_statements)   # comparative only
      output: top_2_variants
      guard: calibration_record_required
    - id: design_test
      set: { MPE, epsilon, max_run_time }    # the three pre-registered params
      choose_method:
        continuous_monitoring: mSPRT
        low_conversion_rate:   sequential_SPRT
        default:               bayesian_decision
    - id: run
      rule: never_call_fixed_horizon_early   # Rule 0
    - id: decide
      ship_if:  expected_loss(variant) < epsilon AND lift > MPE
      kill_if:  expected_loss(control) < epsilon
      else:     inconclusive -> log_and_release_or_extend
    - id: log
      action: record(outcome, params, posteriors)  # feeds meta-learning + SSR calibration
  guardrails:
    - one_variable_per_test_on_low_traffic
    - practical_significance_over_statistical

Meta-learning is the factory's structural edge. No single small-N test is conclusive, but the log of hundreds of them is. Pooling outcomes across experiments (hierarchical / empirical-Bayes priors) lets each new test borrow strength from the history — so the factory's many thin experiments compound into priors that a one-off tester never accumulates.

Provenance Note¶

• Primary-sourced, fetched & verified: peeking 26.1% FPR and the 2.9%/1.0% peek-adjusted thresholds (Miller); the full sequential procedure, thresholds, N=2,922/d=106 and 17.4% savings (Miller); Beta(S+1,F+1) posterior + closed-form Pr(B>A) (Miller). The always-valid-inference mechanism* — a sequential hypothesis-test interface valid at any stopping time — is confirmed from the arXiv:1512.04922 abstract.
• Secondary — search summary, NOT independently fetched (treat as reported, not verified): the ">20%→<5%" production false-positive figure attributed to Optimizely's Stats Engine. The Optimizely whitepaper URL now resolves to a marketing page, so the figure could not be confirmed against a primary. The "mSPRT" name and any specific mixing-parameter constants (e.g. τ) are likewise unverified here — the mechanism claim stands (textbook); the constants do not. A future ingest should fetch the full arXiv:1512.04922 PDF or a primary Stats-Engine source before promoting these to verified.
• Computed for this KB (reproducible from the formulas here): every sample-size figure in §2 (two-proportion formula) and all Bayesian numbers in Example A (200k-draw Monte Carlo over the stated Beta posteriors).
• Framing attributed, not a sourced constant: the expected-loss decision rule and "threshold of caring" (Stucchio, Bayesian A/B Testing at VWO); ε is presented as an operator-set business parameter because no universal value exists. The VWO blog page does not carry the formula — do not cite it for one.

Sources¶

• Evan Miller — How Not To Run An A/B Test
• Evan Miller — Sequential A/B Testing
• Evan Miller — Bayesian A/B Testing
• Johari, Pekelis & Walsh — Always Valid Inference: Bringing Sequential Analysis to A/B Testing (arXiv:1512.04922)
• Optimizely — Stats Engine (the previously-circulated whitepaper URL now redirects to a marketing page; the ">20%→<5%" figure here is reported/secondary, unverified against a primary)
• Chris Stucchio — Bayesian A/B Testing at VWO (VWO SmartStats whitepaper) — expected-loss / threshold-of-caring framing

Concepts¶

Extracted from this source: minimum-detectable-effect · sequential-testing · bayesian-decision-rule · experiment-loop.

Pre-screening and calibration tie-ins: synthetic-consumer-panels · semantic-similarity-rating · embedding-similarity · anchor-statements · calibration-loop · domain-transfer-risk · agent-workflow-pattern.