Creative–landing-page interaction: read all four combinations

Why can a winning creative lose on a different page?

A headline may attract people who need the explanation on one page, while another headline works with a shorter path. When the effect of changing the creative depends on which landing page follows it, the two factors interact. Selecting each asset from a separate average can miss the combination that matters.

This is a proposed diagnostic method, not a report of a test I have run. Start with the one-variable creative and landing-page method when the practical question is simply which single change to investigate. A planned four-combination experiment answers a different question: does the relative performance of the two creatives change between the two destinations?

Use four combinations only when the question needs them

Suppose two compliant messages and two functional landing pages make credible promises about the same offer. A full two-by-two design includes A→1, A→2, B→1 and B→2. Merely changing both assets between Monday and Friday does not create this design: time, auction conditions and inventory may explain the difference.

Do not split scarce traffic into four cells just to make the test look thorough. Confirm that all four combinations are legitimate user journeys, that the available allowance can support the planned analysis, and that each can receive comparable eligible traffic. A broken destination needs repair, not experimental exposure. If a platform cannot provide or document suitable assignment, treat a comparison as observational and describe its confounding limits.

Write the assignment and measurement rules first

Define the eligible audience, placement strata, geography, device, schedule, event, attribution window and maximum exposure before launch. Where feasible, randomize both factors at a defined unit that avoids the same person switching between arms. Record that unit and repeated exposure; an impression count alone does not establish independent observations.

Keep the offer, bidding approach and other settings fixed to the extent the system allows. Log unavoidable delivery or auction differences. Automatic allocation that favors a cell can change the experiment; four campaign names and equal budgets do not prove random assignment. Verify the realized allocation within important traffic slices.

Use the same outcome definition in every cell and allow equivalent maturation. For retail, specify the order status and how cancellations or returns affect it. For an acquisition funnel, keep registrations, deposit events, unique depositors and approved first-time depositors separate. An ad click or an HTTP success response is not an approved business outcome.

A relevant primary measure might be qualified outcomes per assigned eligible unit, with cost per qualified outcome and total spend as economic checks. Choose it for the question; CTR alone does not measure the combined journey. Plan sample size and uncertainty around a commercially meaningful interaction before launch, rather than assuming the illustrative counts below are enough.

A fictional crossover hidden by the average

Invented arithmetic example. These are not James’s, a client’s or a platform’s results, and they are not a significance or sample-size calculation. Assume each combination has 10,000 assigned eligible impressions and an equally mature approved-order observation window. For this arithmetic illustration, each impression represents a different eligible person, assigned before exposure and kept in one arm; each approved order is counted once for that person’s assigned arm. Real repeated exposure requires allocation and analysis at the randomized unit. The counts below illustrate conditional rankings only.

The pooled view gives B a small numerical lead. On page 1, B leads A by 0.16 percentage points; on page 2, B trails A by 0.14 points. The page 2 creative effect minus the page 1 creative effect is −0.14 − 0.16 = −0.30 percentage points. That is the interaction contrast on this outcome-rate scale. It asks whether the creative effect changes with the page; it is not a claim that either difference is statistically reliable.

Pooling also makes page 1 look better overall: 40 orders versus 34 on page 2. Yet A produced more orders with page 2 in this invented table. Picking the highest average creative and the highest average page would therefore discard the question of fit. None of these rates measures profit, customer value or incremental orders.

Read the cells before choosing the next action

Check assignment, delivery mix, tracking and maturity first. Then estimate the creative effect within each page, with uncertainty that reflects the assignment unit. Evaluate the interaction rather than calling it established because one cell is significant and another is not. The apparent crossover can still be sampling variation or a measurement problem.

If supported by adequate evidence and acceptable economics, retain a combination for the context it serves. If uncertainty remains wide at the agreed cap, record an inconclusive result; do not continue spending until a preferred story appears. Where the design is observational, write a hypothesis for a controlled follow-up rather than a causal winner.

A useful handoff names the tested pair, eligible traffic, approved outcome, exposure and cost, uncertainty, changes during the run, and the next review. Keep a safety stop for broken journeys and policy problems, and specify how that stop changes the interpretation of the test.

Method basis and limits

NIST’s full factorial design reference describes crossing factor levels and separating main effects from interactions in a planned design. Applying that structure to a creative and destination is the proposal in this note; the reference does not validate a particular ad platform’s assignment or these fictional numbers.

For a first diagnosis, use placement, creative or landing page: what should change next? For changing traffic composition over time, use creative fatigue versus traffic mix. Those questions should be resolved before interpreting an interaction as a reason to scale.

A decision record you can reuse

Download the plain-text measurement decision brief. It separates the observation, evidence, limits and next check. No account or email is required.

Browse James’s work and case studies, read the career background, or return to this week’s technical notes.