Test the creative–landing-page match, one change at a time

Choose the mismatch you want to resolve

Suppose a push creative attracts clicks but its promise is hard to find on the destination page. I would write one hypothesis: making the creative headline accurately describe the existing landing-page experience will improve qualified outcomes per assigned exposure.

That hypothesis changes the headline only. Keep its image, description, CTA and landing page fixed. If you replace the whole creative, call it a creative-package test: you can compare packages, but cannot attribute the effect to one component.

For pop traffic without a comparable creative exposure, start with a landing-page test on the same arrival path. Do not present pop arrivals and push clicks as an equivalent denominator.

Make the pairing explicit

Run stage 2 as a new comparison after stage 1 closes. If stage 1 is inconclusive, retain C1 as the control rather than silently promoting C2. Comparing C1 + L1 against C2 + L2 changes two factors at once.

This sequence estimates each change conditional on the asset held fixed. It does not identify every creative–lander interaction or prove the globally best pairing. A properly designed factorial experiment can investigate interactions when traffic and budget support it.

Write the test brief before allocating traffic

1. Define a business outcome and its denominator

   For a creative test, use an exposure-based denominator defined before the click when available. Comparing only clickers can select different populations after the creative changes. For a lander test, randomize eligible arrivals before rendering either page and retain all assigned units in the analysis.

   Choose an outcome such as approved FTD per unique assigned unit when the contract and instrumentation support it. Keep cost per approved FTD as an economic guardrail. If exposure-level measurement is unavailable, document the limit and treat the creative comparison as an operational signal.

2. Specify assignment and fixed controls

   Use concurrent random allocation with a documented split and persistent assignment for returning visitors where permitted. Hold source, GEO, device targeting, placements, bid, frequency cap, offer, attribution settings and event mapping steady. Check assignment counts and eligibility before interpreting outcomes.

   Two separate campaigns may buy different auctions. Equal budgets or a nominal 50/50 rotator do not by themselves prove comparable exposure. If the source adaptively reallocates creatives and you cannot obtain randomized assignment, label the result observational.

3. Predefine evidence and spending limits

   Choose the minimum useful effect, baseline rate, analysis method, sample requirement and review date before launch. Include confidence intervals for the planned comparison. Set a total test cap, daily pacing and pending-outcome exposure limit. If the budget cannot support the sample, narrow the question or accept an inconclusive outcome.

4. Allow equal maturation

   Choose a conversion window and a separate approval cutoff from observed delays and partner rules. Compare acquisition cohorts at the same age. A deposit event is an action, a unique depositor is a person in a period, an FTD is a first qualifying deposit, and an approved FTD has passed validation. These counts cannot substitute for one another.

5. Define early stops without chasing a winner

   Stop for broken links, tracking loss, a material quality breach or the agreed spending cap. Record the reason. Do not stop at the first attractive CPA or keep extending a test until it looks positive. A fixed-horizon design waits for its planned endpoint; repeated formal decisions require a prespecified sequential method.

NIST describes random assignment for experiments with one primary factor. The practical brief here applies that principle to campaign comparisons; it is not a platform-specific guarantee of random delivery. NIST: completely randomized designs.

Worked example: better clicks, no clear winner

Assume a controlled push test can randomly assign unique eligible users before exposure and preserve assignment. C2 changes only the headline; both arms use L1. The planned primary outcome is approved FTD per assigned user. The fictional cap is $300 total, with an equal $150 allocation per arm and a $30 CPA guardrail.

The observed outcome difference is 0.01 percentage points, or one additional approved FTD per 10,000 assigned users. C2 drew more clickers, but seven versus six approved outcomes is sparse evidence. Both point estimates pass the fictional cost guardrail; that alone does not establish a winner.

The total cap is exhausted. I would close this run as inconclusive unless the prespecified analysis supported a useful difference, retain C1 and document the headline hypothesis for a future adequately sized test. Do not buy more traffic merely to rescue the result. If an independently justified lander test follows, use C1 in both arms.

For sparse binary outcomes, avoid a simplistic normal interval that can imply impossible negative rates. NIST discusses Wilson and exact binomial intervals for a single proportion. A decision between arms requires an appropriate interval or test for their difference; overlapping individual intervals are not a substitute. NIST: proportion confidence intervals.

Carry forward the conclusion the design supports

Save the two assets, version IDs, assignment rule, event contract, cohort cutoff, spend, analysis and decision. Record imbalance, tracking losses or concurrent source changes. A compromised assignment may invalidate the comparison even if its dashboard looks precise.

A valid randomized comparison can estimate the relative effect of the tested variants within the experiment. It does not establish advertising incrementality against no advertising. Attributed deposits are not revenue, and a lower acquisition cost does not establish lifetime profitability. Those require different evidence.

Continue the method

• The earlier campaign comparison framework
• Bound the spend behind a placement decision
• Resolve missing source conversions before testing

All technical notes →

Continue the investigation

If the creative ranking may depend on the destination, read the four-combination interaction method.