Method demonstration. This method demonstration uses fictional examples. It does not report client results or independently verified production behaviour.
Start with a matched comparison
Before replacing a creative, compare its performance within the same placements, devices and reporting windows. Then check what share of delivery each segment received. A campaign average can fall even when every measured segment performs exactly as before.
This method applies to campaigns with identifiable creatives and usable impression and click reports, such as push advertising. For pop traffic without a comparable impression-to-click creative interaction, use the relevant landing or downstream event measures instead. Do not carry a push CTR rule into a different ad format.
The practical question is whether a creative change is the next useful test. This is a diagnosis of competing explanations, not proof of why users changed their behaviour.
Write down what must remain comparable
Export two complete reporting windows with the same timezone and comparable operating hours. Save the creative version, landing-page version, placements, device and browser mix, targeting, bids and any schedule changes. A renamed ad is not necessarily the same asset.
For downstream comparisons, define the event and allow both acquisition cohorts the same time to mature. A registration, a deposit event and an approved first-time depositor are different outcomes. Recent clicks with unfinished approvals cannot be compared fairly with an older settled cohort.
Start with a few segments large enough to inspect. Splitting into dozens of tiny cells can replace one misleading average with many unstable rates. Keep newly added placements separate where there is no earlier comparison.
Worked example: a CTR decline without a segment decline
Fictional teaching example. These are invented impressions and clicks, not James’s campaign results. The same creative and landing page run in both windows. A and B are anonymous placement groups.
| Segment | Earlier window | Later window |
|---|---|---|
| Group A | 80,000 impressions; 1,600 clicks; 2.0% CTR | 30,000 impressions; 600 clicks; 2.0% CTR |
| Group B | 20,000 impressions; 200 clicks; 1.0% CTR | 90,000 impressions; 900 clicks; 1.0% CTR |
| Total | 100,000 impressions; 1,800 clicks; 1.8% CTR | 120,000 impressions; 1,500 clicks; 1.25% CTR |
The aggregate CTR falls from 1.8% to 1.25%, a 30.6% relative decline. Group A’s share of impressions falls from 80% to 25%. Neither segment’s CTR changes.
Reweight the later segment rates using the earlier impression shares: (80% × 2.0%) + (20% × 1.0%) = 1.8%. This descriptive calculation shows that the observed mix change accounts for the aggregate CTR difference in this simplified example. It does not estimate what would happen if the old mix could be bought again, or prove that users within each group stayed the same.
I would investigate why delivery moved toward B before ordering a creative refresh. There are no costs or approved outcomes in this example, so it cannot identify which placement is more profitable or justify a bid increase.
Use the pattern to choose the next check
| Observed pattern | Next check | Bounded next action |
|---|---|---|
| Only the aggregate rate declines | Compare delivery shares and within-segment rates. | Investigate the changed allocation; do not label the creative fatigued. |
| Comparable segments decline for the same asset | Check exposure history if available, destination changes, delivery conditions and event collection. | Treat fatigue as one hypothesis; prepare one creative variant with fixed controls. |
| Clicks hold but approved outcomes fall | Align cohort age, landing behaviour and approval rules. | Resolve the downstream change before declaring the ad a failure. |
| Sparse data or different segments | State what cannot be matched and how uncertain the rates are. | Keep an exposure cap; collect a useful comparison or close as inconclusive. |
Repeated exposure can support a fatigue hypothesis only when the source actually provides a meaningful frequency or exposure history. Days since launch alone do not tell you how often an individual saw an ad. Do not invent user-level frequency from campaign totals.
Test a refresh without losing the explanation
Write one hypothesis, for example: a new headline improves the chosen mature outcome within the existing traffic slice. Keep the image, destination, offer and other controllable settings fixed. Use randomized allocation where available; otherwise describe the allocation limitation explicitly. Decide the spend limit, outcome definition, analysis and stopping rule before starting.
Evaluate the primary outcome and its uncertainty, not just the prettiest CTR. If the refresh draws more clicks but fewer qualified actions per exposure, it has not met the original objective. If the sample is too small within the agreed cap, report an inconclusive test rather than inventing a winner.
Save the asset versions, segment export, weighting calculation and decision. A useful handoff says which explanation remains plausible and what would change the decision.
For a complete experiment brief, continue to testing the creative–landing-page match one change at a time. The placement, creative and lander framework explains how these decisions fit together.
What this method can and cannot establish
This note uses arithmetic and a proposed review method, not a completed experiment or measured improvement. The example controls only the two displayed segments; real campaigns may contain changes inside each group. A matched observational comparison narrows an investigation but cannot prove a causal fatigue effect. A valid randomized test answers its specific variant question; it does not establish advertising incrementality or customer lifetime value.