Incrementality Testing

What It Is

Incrementality testing answers the counterfactual question attribution cannot: what would have happened without this spend? It randomly withholds a channel/campaign from a control group and measures the difference. The result is incremental conversions — those genuinely caused — not merely correlated.

Lift % = (Test conversions − Control conversions) / Control conversions

A channel can look dominant in marketing-attribution yet near-zero in incrementality (e.g. retargeting buyers who'd already convert). Only the controlled withhold reveals true cause.

The Methods Ladder

All are RCTs; they differ in the randomization unit:
- User-level holdout (Meta/Google Conversion Lift) — randomize individuals into ad-eligible vs withheld. Purest, where the platform supports it.
- PSA / ghost ads — control sees a placebo PSA, or (ghost ads) the system logs the ad the control would have seen. Ghost ads give a clean counterfactual without paying to serve placebo creative.
- Geo-experiment — randomize markets when you can't randomize users (privacy, offline conversions). See geo-experiment.

How It Applies to Marketing Factory

Incrementality is the causal-measurement layer that lets the factory allocate budget on truth rather than credit. It is the budget-level extension of the experiment-loop (tactic tests optimize within a channel; incrementality optimizes across them), it produces incremental-roas as its decision metric, and — because tests have few units and run for weeks — it inherits small-N discipline (pre-registered thresholds, no peeking).

Referenced from: incrementality-and-geo-experiments