Pop-Up Timing on Category Pages: A/B Test Idea

Page exits right after the pop-up appears, session recordings of visitors closing it without reading, or a low response rate can each mean that the pop-up on the listing page either interrupts visitors or goes unnoticed. If timing is what matters, one trigger brings more purchases across all assigned visitors without raising page exits. Before launch, make sure the pop-up does not cover the full screen on mobile and the close icon stays visible and easy to tap; if the problem is the offer itself, timing will not fix it.

Purchase conversion rate

Compare purchases among all eligible visitors assigned before the pop-up trigger, including visitors who never see it.

Also track

• Pop-up response rateReport reach and response separately because the two triggers may expose different numbers of visitors.
• Pop-up CTA click-through rateMeasure clicks on the pop-up's primary button.
• Page exit rateCompare exits from pages where the pop-up was eligible to appear.
• Session durationUse session duration only as context; longer sessions are not automatically better.

• TriggerCompare display on the first eligible page view with display after the visitor has scrolled 50% of that page in the same session.
• Keep constantUse the same offer, design, size, position and frequency cap.
• AssignmentAssign visitors before the trigger; keep those who never see the pop-up in the primary analysis.
• Primary outcomeReview purchases and exits alongside pop-up interactions.
• Separate testsTest the offer, design and frequency cap separately.

1. 1The arm with the higher pop-up response rate or pop-up CTA click-through rate is not automatically the winner, because the two triggers reach different numbers of visitors. Compare purchases across all eligible assigned visitors, including those who never saw the pop-up, and report reach and page exits alongside.
2. 2Timing should be the only difference: no other pop-up should fire in the same session, and the frequency cap should keep this one from reappearing on every page. If either condition broke during the test, the trigger cannot take credit for the difference.
3. 3This offer, this design and category pages set the bounds of the result. Even if on-load display wins, do not roll it out to pages outside the test automatically; try each new context on its own and keep the device results apart.
4. 4Because the scroll trigger reaches fewer visitors, the purchase gap between the arms may stay small. Run the gap through the A/B test significance calculator using all eligible assigned visitors, not only those who saw the pop-up; if it is not significant, record the test as inconclusive and leave offer or design changes to a separate experiment.