Method demonstration. This is a proposed method, not verified client results. Worked examples are fictional.
Choose the question before opening the report
A promotion can generate plenty of attributed orders without showing how many extra orders the advertising created. Before renewing the same spend, separate two questions: did the orders produce acceptable contribution, and would those orders have happened without the intervention?
The retail ROAS, margin and stock review handles the first question. This note handles the second: choosing a comparison that can support a narrower incrementality claim. It is a proposed evaluation method, not a report of a promotion or lift study I ran.
Attribution assigns credit under a reporting system's rules. Incrementality asks about the additional outcome caused by an intervention. Google's conversion-lift documentation explicitly distinguishes attributed conversions from incremental conversions measured against a control. That distinction is useful here; it does not imply access to Google's tools or experience operating them. Google: attributed and incremental conversions.
Define exactly what the control will miss
Write one sentence: “We want to estimate the effect of adding [intervention] to [existing conditions] for [eligible population] during [period].” A discount, extra advertising and a product-page redesign are three different interventions.
If both groups can use the same discount but only one receives the added advertising, the comparison concerns that advertising during the promotion. It cannot isolate the discount's effect. If advertising and discount change together, the result concerns the package. Do not rename that package effect “media lift.”
Define the unit of assignment before exposure: a permitted customer identifier, household or suitable geographic unit. Use a design appropriate to that unit and analyze everyone assigned, including people who never click. Random assignment is a design property, not something established by equal budgets. NIST describes random assignment of treatment levels to experimental units. NIST: completely randomized designs.
Match the decision to the available evidence
| Available evidence | What it supports | Next decision |
|---|---|---|
| Attributed orders and reconciled costs only | Recorded order economics under the stated attribution rules | Apply the agreed spending boundary; leave incremental return unknown. |
| Sales before and during a promotion | A descriptive change across periods | Investigate seasonality, pricing and stock changes; do not credit the entire change to advertising. |
| Concurrent randomized groups with reliable assignment and complete outcomes | An estimate of the specified intervention's effect within the tested population and period | Evaluate uncertainty and incremental contribution against the predefined decision rule. |
| Control contamination, missing orders or insufficient precision | A limited or inconclusive comparison | Document the failure and decide whether a separately funded redesign is worthwhile. |
For geographic tests, independent units are geographies, not every order inside them. A small number of regions needs a suitable design and analysis. Do not treat thousands of purchases as thousands of independently randomized customers.
Fictional example: more orders, weaker incremental contribution
Every number below is invented. Assume two equal, randomly assigned groups of 10,000 eligible customers. Both can access the same promotion. The treatment group is eligible for an additional advertising intervention; the control is withheld from that intervention. Other conditions are held stable, assignment is retained, and the same return cutoff applies.
| Measure | Added-advertising group | Control group |
|---|---|---|
| Eligible assigned customers | 10,000 | 10,000 |
| Completed orders | 1,200 | 1,000 |
| Net product revenue | $60,000 | $50,000 |
| Contribution before additional advertising | $18,000 | $15,000 |
| Additional advertising cost | $4,000 | $0 |
The observed difference is 200 orders and $10,000 net revenue. The incremental revenue-to-spend point estimate is $10,000 ÷ $4,000 = 2.5×. Yet the contribution difference after additional advertising is $18,000 − $15,000 − $4,000 = −$1,000.
Here contribution already includes discounts, returns and agreed variable order costs, so those costs must not be deducted again. This is contribution, not company net profit. With unequal group sizes, compare appropriately normalized outcomes; subtracting raw totals would be misleading.
These aggregates illustrate arithmetic, not statistical certainty. A real analysis needs the prespecified uncertainty estimate using the assignment unit and outcome distribution. The example neither proves a loss nor supports expansion: its point estimate misses the economic objective, and its precision is unknown.
Protect the comparison through the promotion
Before launch, record eligibility, assignment, intervention versions, minimum useful effect, required precision, sample plan, spend cap and decision date. Define net revenue and contribution consistently. Preserve equal observation time and a stated return allowance. Missing outcomes must remain missing, not become zero.
Check whether control customers receive the intervention through another route, whether stock constraints affect one arm, and whether coupon sharing crosses the boundary. Record operational fixes rather than hiding them. If attribution is the only available reporting view, do not assume it captures every assigned customer's outcomes.
Include a predefined follow-up period where practical to examine whether an apparent gain is followed by weaker demand. Inspect the agreed product range so movement between products is not mistaken for additional business. These are proposed checks, not evidence that displacement occurred.
Leave a decision another buyer can use
The handoff should name the intervention, comparison population, outcome basis, uncertainty, contribution estimate and next authorized action. A useful conclusion may be “economically unpromising at this estimate,” “inconclusive,” or “worth a bounded replication.” It need not name a winner.
If a valid control is unavailable, use reconciled economics to manage exposure and state that incrementality remains unknown. Continue with budget pacing while outcomes mature or the media buyer field guide.
A decision record you can reuse
Download the plain-text measurement decision brief. It separates the observation, evidence, limits and next check. No account or email is required.
Browse James’s work and case studies, read the career background, or return to this week’s technical notes.