Calibration Loop

What It Is

A calibration loop closes the gap between a tool's predictions and reality by logging both, every cycle, and bounding trust by the observed hit rate. For synthetic-consumer-panels: record which variant the panel picked, then record the real post-publication metric (clicks, replies, captures), and track how often the panel was right.

The Rules

  • Mandatory, not optional — without it the panel is an unvalidated guess (see domain-transfer-risk).
  • Advisory until calibrated — at 2–3 published experiments/week it takes ~6–10 weeks before the synthetic-vs-real hit rate is statistically meaningful. Until then the panel is a tiebreaker, never a veto.
  • Real behavior > stated intent — the factory's ground truth (actual engagement) is better than the survey panels SSR was validated against, which only approximate behavior.
  • The log is an asset — calibration data is itself publishable original research (the first-party-statistics GEO play) and the AI-vs-manual benchmark the KB's gaps name.

How It Applies to Marketing Factory

Every predictive agent gate needs a calibration loop wired into its agent-workflow-pattern Output: the structured experiment row is exactly the data the loop consumes. This generalizes beyond SSR — any time an agent makes a call that reality will later score (subject lines, send times, channel picks), log prediction-vs-outcome and let the measured hit rate set how much autonomy the agent gets.

Referenced from: ssr-synthetic-panels