Incrementality & Geo-Experiments — Causal Measurement for Budget Allocation

For factory operators. Attribution tells you which touchpoints correlated with conversions; it cannot tell you which spend caused them, and it cannot tell you where the next dollar should go. This document is the causal-measurement layer: how to run experiments that isolate the incremental revenue a channel produces, and turn that into a budget-allocation decision. It is the natural completion of the experiment-loop — tactic tests optimize a page; incrementality tests optimize the budget.

The thesis: stop asking "what did this channel get credit for?" (attribution) and start asking "what would have happened without it?" (incrementality). The gap between the two is often enormous — much branded search, retargeting, and "last-click" credit is conversions that would have happened anyway.

1. Why Incrementality (the hole attribution leaves)

The KB already documents that marketing-attribution observes correlation, not causation and recommends holdout tests for the causal signal no model provides. Incrementality is how you run those holdouts. The core question is counterfactual: incremental conversions = conversions with the channel on − conversions that would have happened with it off.

A channel can score huge in attribution and ~zero in incrementality (e.g. retargeting people who were already going to buy). Only a controlled experiment — withholding the ad from a randomized control — reveals the true causal lift.

2. The Methods Ladder (cleanest → most practical)

All three are randomized controlled trials; they differ in what gets randomized.

Universal lift formula:

Lift % = (Test conversions − Control conversions) / Control conversions
Method Randomization unit Clean? Cost / constraint
User-level holdout (Meta/Google Conversion Lift) individual users Purest RCT Needs platform support; control users withheld from ads
PSA / Ghost ads individual users; control sees a placebo (PSA) or would-have-been ad (ghost) Very clean PSA costs money to serve non-branded ads; ghost ads avoid that by logging the ad the control would have seen
Geo-experiment geographic regions Robust, platform-agnostic Needs many comparable geos; coarser, slower
  • User holdouts are the gold standard where the platform offers them (Meta Conversion Lift, Google Conversion Lift): randomly split the target audience into eligible-to-see-ads vs withheld, measure the lift.
  • Ghost ads solve the PSA cost problem: instead of paying to show the control a placebo, the system records the ad the control user would have been served, giving an apples-to-apples counterfactual without wasted spend.
  • Geo-experiments are the answer when you can't randomize users (privacy walls, offline conversions, whole-funnel channels): randomize markets instead.

3. Geo-Experiments — the Geo-Based Regression (GBR) Model

When user-level randomization isn't available, randomize geographies. The methodology (Vaver & Koehler, Measuring Ad Effectiveness Using Geo Experiments, Google Research; operationalized in Google's "Estimating causal effects using geo experiments"):

Design. Partition the country into non-overlapping geos, then form control/treatment groups by stratified randomization (strata of similar-size geos) or matching (pick geos predictive of each other) when geos are few or heterogeneous. Rule of thumb: ≥30 geos.

Phases. Pretest (4–8 weeks, identical campaigns everywhere) → Test (3–5 weeks, treatment geos get the spend change) → optional cool-down (for delayed/offline conversions).

The model. Predict the counterfactual from control geos, then fit a weighted linear regression across geos:

y₁,ᵢ = β₀ + β₁·y₀,ᵢ + β₂·δᵢ + εᵢ
  • y₁,ᵢ = test-period response in geo i; y₀,ᵢ = pretest response (controls for trend/seasonality)
  • δᵢ = incremental ad spend in geo i = observed − counterfactual spend
  • β₂ = iROAS — the parameter of interest. β₂ = 3.1 means each $1 of incremental spend caused $3.1 of incremental revenue.
  • Weights 1/y₀,ᵢ handle heteroscedasticity across differently-sized geos.

Precision is reported as a CI on iROAS (e.g. "iROAS ± 1.0 at 95%"); it improves with larger spend differences, bounded by inventory.

4. iROAS Is the Budget-Allocation Decision Metric

Blended ROAS (revenue / spend) double-counts organic demand and over-credits bottom-funnel channels. Incremental ROAS answers the only question that matters for allocation: what did the marginal dollar cause? The allocation rule:

Shift budget toward channels with the highest marginal iROAS until marginal iROAS equalizes across channels (or hits your target ROAS floor). A channel with blended ROAS 8× but iROAS 0.9× is destroying margin at the margin — cut it.

This is the decision the experiment-loop cannot make from tactic tests alone: A/B tests optimize within a channel; incrementality optimizes across channels.

5. Geo-Experiments Are Small-N Experiments

A geo-test has ~30–200 units (geos), not thousands — it lives squarely in minimum-detectable-effect territory. The same discipline applies:

  • MDE on geos: with few, noisy geos you can only detect large iROAS differences; compute whether your spend change is big enough to clear the CI before running.
  • Don't peek the running geo-test as if fixed-horizon; use sequential-testing / always-valid methods, or decide with a bayesian-decision-rule (posterior over iROAS, expected loss of reallocating < threshold).
  • Pre-register the spend change, test window, and decision threshold — same three parameters as the experiment loop.

6. Factory Integration

Incrementality is the budget-layer extension of the experiment-loop: a recurring causal test that re-estimates each major channel's iROAS and feeds the allocation decision. Most of it is agent-ownable — forming matched geo groups, applying the spend change via geo-targeting, fitting the GBR model, and reporting the iROAS CI. The human decision is the reallocation itself. Run quarterly per major channel; treat the iROAS estimates as the empirical-Bayes priors the experiment loop already calls for.

Provenance Note

  • Primary-sourced, fetched & verified: the GBR model y₁=β₀+β₁y₀+β₂δ+ε, β₂=iROAS interpretation, stratified-randomization/matching, pretest/test/cool-down phases, and the ≥30-geo rule (Google, Estimating causal effects using geo experiments, unofficialgoogledatascience.com).
  • Originating paper (not fetched as text — PDF was binary): Vaver & Koehler, Measuring Ad Effectiveness Using Geo Experiments (Google Research). Cited as the method's origin; operational detail above comes from Google's methodology blog.
  • Standard/established (not a contested claim): the lift formula (Test−Control)/Control; user-holdout, PSA, and ghost-ads definitions (Meta/Google Conversion Lift; Johnson, Lewis & Nubbemeyer, Ghost Ads).

Sources

Concepts

Extracted from this source: incrementality-testing · geo-experiment · incremental-roas

Related concepts: marketing-attribution · experiment-loop · minimum-detectable-effect · bayesian-decision-rule