Synthetic Consumer Panels via SSR — what's validated, what's not, how the factory uses it

The primary source

Paper: "LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings" — arXiv:2510.08338, PyMC Labs (Maier, Wiecki) + Colgate-Palmolive, Oct 2025. Validation data: 57 personal-care product surveys, 150–400 human participants each, 9,300 human responses total. (Full text read 2026-06-12; details below are paper-verified, not abstract-inferred.)

Full text in this KB: raw/2026-06-12-ssr-paper-arxiv-2510-08338.md (pandoc conversion of the arXiv v3 HTML). Implementers: Appendix C.1 has the actual reference statement sets to adapt; C.2 has the SSR math.

The verified claims:
- Asking LLMs directly for numeric ratings produces unrealistic response distributions — the naive approach fails.
- Semantic Similarity Rating (SSR): elicit textual reactions from the synthetic respondent, then map text onto Likert distributions via embedding similarity to reference statements.
- SSR achieves ~90% of human test-retest reliability (i.e. ~90% of the way to how consistent real humans are with their own answers on retest — NOT "90% accuracy"; the popular framing overstates it).
- Realistic response distributions (KS similarity > 0.85) + qualitative feedback explaining each rating.

Provenance note: the viral X thread (@HowToAI_, 2026-06) framed this as "90% accuracy / market research is dead." The thread author self-corrected in a follow-up tweet. Use the paper's framing, not the thread's.

Implementation details (from the full paper)

  • Three strategies compared. Direct Likert Rating (ask for a number): ρ≈80% correlation attainment but badly unrealistic distributions (KS 0.26–0.39; models regress to a "safe 3"). Follow-up Likert Rating (a second LLM instance acting as a "Likert rating expert" scores the free text): intermediate. SSR wins on both distribution similarity and concept ranking.
  • Models: GPT-4o and Gemini-2.0-flash for production runs — chosen for response consistency after trials with gemini-1.5-flash, gemini-2.5, and o3. Temperature 0.5 (1.5 made little difference), top_p 0.9. Implication: frontier-best is not required; consistent mid-tier models suffice.
  • Sampling: n=2 responses per synthetic respondent was sufficient for stable results. (Not 10 — this cuts naive cost estimates ~5×.)
  • Anchors: six different reference-statement sets, PMFs averaged across them (App. C.1). Don't build one anchor set; build several and average — this is the robustness trick.
  • Embeddings: OpenAI text-embedding-3-small, cosine similarity; text-embedding-3-large changed nothing. Embedding cost is negligible.
  • Personas are load-bearing, not garnish: the paper states the results "are only achieved when LLMs are prompted to consider demographic attributes of a person they are being asked to impersonate." Age and income response patterns mirrored real data. Skipping persona conditioning breaks the method.
  • Known bias found: for less-attractive concepts, LLMs rate lower than humans on average — synthetic panels are not uniformly agreeable; they exaggerate at the bottom of the scale too.
  • Authors' own caveat: validity "depends on the alignment between training data and the survey domain," and results depend on embedding-model choice. (Direct support for the domain-transfer caution below.)

What this is for in the marketing factory

The factory's core design problem (per projects/agentisable/PLAN-2026-06-11-marketing-factory.md): marketing has no "tests pass." SSR offers a pre-publication test layer — a wind tunnel for drafts:

Agent workflow pattern (trigger → input → action → output):
1. Trigger: a draft (post, page, email, positioning statement) passes the voice/fact gates.
2. Input: the draft + 5–10 synthetic personas built from real buyer profiles (e.g. the teardown PRD's personas: seed-stage founder, dev-shop owner, forwarded-to CTO).
3. Action: each persona produces a textual reaction (never a direct score); SSR maps reactions to Likert distributions against reference statements (clarity, relevance, intent-to-act).
4. Output: comparative ranking of variants + the qualitative reactions; winner proceeds to publication; result logged as an experiment row.

Hard limits (write these into any implementation)

  1. Domain transfer is unproven. Validation is purchase intent for personal-care CPG. Developer/founder audiences — skeptical, contrarian, HN-flavored — are plausibly the worst case for synthetic-panel agreeableness bias. No published validation exists for dev-audience marketing (as of 2026-06).
  2. Comparative use only. Rank variant A vs. B. Never treat an absolute score as a prediction of real-world performance.
  3. Calibration is mandatory. Log synthetic-panel picks vs. real post-publication metrics every cycle. Trust the panel only as far as its observed hit rate. This calibration data is itself publishable original research ("do synthetic panels predict developer-audience engagement?") — the first-party-statistics GEO play; nobody has published this for dev audiences.
  4. Known biases: naive LLM elicitation is under-dispersed (the "safe 3" regression); SSR fixes the distribution problem. Directional bias is not simple agreeableness — the paper found LLMs rate less-attractive concepts lower than humans do. Expect systematic distortion at both ends of the scale; this is another reason for comparative-use-only and calibration. (Corrected 2026-06-12 after full-paper read; an earlier version of this file claimed uniform agreeableness bias — the paper's data is more nuanced.)

Cost: setup and running (estimates marked as such)

Running cost — LLM generation tokens are ~99% of marginal compute, and it rounds to zero. Per response ≈ 1,500 tokens in / 250 out → ~$0.003–0.008 depending on model class. With the paper's n=2 sampling: 10 personas × 2 samples × 2 variants = 40 responses ≈ $0.12–0.35 per A/B test (estimate). Embeddings: under a cent. The paper's 9,300-response human dataset would cost ~$30–90K at panel-vendor rates ($3–10/complete); the synthetic equivalent is under $100 — the ~500–1000× ratio is the real story behind the viral framing.

Setup cost — no human panel required. The paper's 9,300 human responses were for scientific validation of the method; consumers of the method don't re-run that. The factory's ground truth is better than a survey panel anyway: real post-publication behavior (clicks, replies, captures), which stated-intent surveys only approximate. Setup is:

  1. Personas — agent labor from data already held (audit conversations, prospect replies, community comment patterns). Paper-backed requirement: demographic/attitudinal conditioning is load-bearing (§ above).
  2. Anchor statement sets — the real intellectual work: ~5–7 statements per scale level × several sets (paper used six, averaged). A few hours of careful writing + operator review. No standardized dev-audience scale language exists; this is written from scratch.
  3. Pipeline code — one small factory build (gate on the existing exhaust pipeline): single-digit dollars of compute by current build economics.
  4. Calibration — free in money, expensive in calendar: at 2–3 published experiments/week, ~6–10 weeks before the synthetic-vs-real hit rate is statistically meaningful. Until then the panel is advisory only — a tiebreaker, never a veto.

Optional accelerator: a small human spot-check (100–200 dev-profile respondents via Prolific-class panels, ~$300–1,500) compresses validation into a week. Skip at v1: real metrics arrive fast enough at factory volume, and "calibrated against real engagement" is the publishable research asset; "calibrated against Prolific" is not.

Where it slots

  • KB: fills Known Gap #1 (explicit agent workflow pattern above) and Gap #2 (the calibration loop generates exactly the AI-vs-manual benchmark data the gap names).
  • Factory implementation: v1 experiment-loop gate (NOT v0 — v0 stays the minimal content-exhaust pipeline). Effort: small; the paper hands over the method.

Concepts

Extracted from this source: synthetic-consumer-panels · anchor-statements · agent-workflow-pattern · domain-transfer-risk · calibration-loop

Full-text primary source: ssr-paper-arxiv-2510-08338