# Pop-Up Timing on Category Pages: A/B Test Idea

Page: https://alidemirbas.com.tr/en/lab/ab-test-library/tests/pop-up-timing-on-category-pages-ab-test-idea

## Hypothesis

A pop-up shown on page load may reach more visitors but interrupt them just as they start their task. One shown after scrolling may reach fewer people but catch visitors who are already engaged with the page. If the cost of interrupting visitors outweighs the extra reach, the scroll trigger could come out ahead on purchase conversion rate; if it does not, the on-load trigger could.

## When to run this test

Page exits right after the pop-up appears, session recordings of visitors closing it without reading, or a low response rate can each mean that the pop-up on the listing page either interrupts visitors or goes unnoticed. If timing is what matters, one trigger brings more purchases across all assigned visitors without raising page exits. Before launch, make sure the pop-up does not cover the full screen on mobile and the close icon stays visible and easy to tap; if the problem is the offer itself, timing will not fix it.

## Primary metric

**Purchase conversion rate**: Compare purchases among all eligible visitors assigned before the pop-up trigger, including visitors who never see it.

## Other metrics

- **Pop-up response rate**: Report reach and response separately because the two triggers may expose different numbers of visitors.
- **Pop-up CTA click-through rate**: Measure clicks on the pop-up's primary button.
- **Page exit rate**: Compare exits from pages where the pop-up was eligible to appear.
- **Session duration**: Use session duration only as context; longer sessions are not automatically better.

## What to check before and during the test

- **Trigger**: Compare display on the first eligible page view with display after the visitor has scrolled 50% of that page in the same session.
- **Keep constant**: Use the same offer, design, size, position and frequency cap.
- **Assignment**: Assign visitors before the trigger; keep those who never see the pop-up in the primary analysis.
- **Primary outcome**: Review purchases and exits alongside pop-up interactions.
- **Separate tests**: Test the offer, design and frequency cap separately.

## What to watch when reading the result

- The arm with the higher pop-up response rate or pop-up CTA click-through rate is not automatically the winner, because the two triggers reach different numbers of visitors. Compare purchases across all eligible assigned visitors, including those who never saw the pop-up, and report reach and page exits alongside.
- Timing should be the only difference: no other pop-up should fire in the same session, and the frequency cap should keep this one from reappearing on every page. If either condition broke during the test, the trigger cannot take credit for the difference.
- This offer, this design and category pages set the bounds of the result. Even if on-load display wins, do not roll it out to pages outside the test automatically; try each new context on its own and keep the device results apart.
- Because the scroll trigger reaches fewer visitors, the purchase gap between the arms may stay small. Run the gap through the [A/B test significance calculator](https://alidemirbas.com.tr/en/calculators/ab-test) using all eligible assigned visitors, not only those who saw the pop-up; if it is not significant, record the test as inconclusive and leave offer or design changes to a separate experiment.

## Frequently Asked Questions

### Why not analyze only visitors who saw the pop-up?

Different triggers expose different populations. Assign before triggering and retain unexposed assigned users in the primary comparison. Use purchase conversion across all eligible assigned visitors.

### Does a higher pop-up response rate identify the winner?

Not alone. Separate reach from response after exposure. Monitor page abandonment as a guardrail because a persuasive interruption can still harm the main task.

### Should a pop-up appear immediately or after scrolling?

Immediate display reaches more visitors but can interrupt them. Scroll-triggered display reduces reach. On mobile, an opening overlay covering the content can also affect search visibility. Keep dismissal visible.