Factory Experiment Loop¶

What It Is¶

A machine-readable loop that turns "should we ship this variant?" into a governed, repeatable process an agent can execute. It assembles the small-N toolkit into one pipeline:

1. Feasibility gate — compute the minimum-detectable-effect from traffic and max run time. If the plausible effect is below it, route to the synthetic path instead of a doomed live test.
2. SSR pre-screen (optional, when >2 variants) — rank variants with synthetic-consumer-panels against anchor statements (comparative use only), keep the top 2. Requires a calibration-loop record.
3. Design — pre-register three parameters: MPE (minimum practical effect worth shipping), ε (threshold of caring), max run time. Choose method by context: always-valid mSPRT for continuous monitoring, sequential-testing (SPRT) for low conversion rates, bayesian-decision-rule as default.
4. Run — never call a fixed-horizon test early (Rule 0).
5. Decide — ship when expected loss of the variant < ε and lift > MPE; kill when the control's loss < ε; otherwise inconclusive.
6. Log — record outcome, parameters, posteriors. This feeds both meta-learning and SSR calibration.

Pre-Registration Is the Discipline¶

Every experiment fixes MPE, ε, and max run time before launch and logs them. No declared stop rule → no test. Pre-registration is what makes an agent's continuous monitoring safe rather than a peeking machine.

Meta-Learning Is the Edge¶

No single small-N test is conclusive — but the log of hundreds is. Pooling outcomes across experiments (hierarchical / empirical-Bayes priors) lets each new test borrow strength from history, so the factory's many thin experiments compound into priors a one-off tester never accumulates. Volume of experiments, not volume of traffic per experiment, is the moat.

How It Applies to Marketing Factory¶

This is the operational spine of the v1 experiment system: it is the agent-workflow-pattern specialised for experimentation, with the synthetic-consumer-panels pre-screen and calibration-loop wired in as first-class steps. It lets a factory running 2–3 experiments/week make valid, low-regret, auditable ship/kill calls instead of guessing.

Related Concepts¶

• minimum-detectable-effect — the loop's entry gate
• sequential-testing — one method branch (continuous / low-rate)
• bayesian-decision-rule — the loop's default decide step
• synthetic-consumer-panels — the optional pre-screen stage
• calibration-loop — keeps the synthetic stage honest
• agent-workflow-pattern — the loop is a specialised instance of this

Referenced from: small-n-experiment-design