Promo Message on the App Loading Screen A/B Test Idea

If the loading screen already stays up for a while, it may be worth using that moment for the campaign, especially when promoted categories get few visits, session recordings skip campaign areas, or shoppers tell support they learned of the campaign late. If the message gets noticed, visitors who see it move on to promoted categories and products more often, and that interest carries through to purchases. Do not launch unless the campaign is live and the message states its terms accurately, and measure first that heavy imagery does not slow loading.

Purchase conversion rate

Compare purchases among eligible visitors assigned to each loading-screen message; do not restrict analysis to visitors who waited long enough to see it.

Also track

• Post-load CTA click-through rateTrack CTA clicks, but do not use them as a substitute for purchases.
• Campaign product purchase rateCompare purchases of promoted products among all eligible assigned visitors.
• Page exit rateCompare exits during loading in both arms.
• Time to main content readinessMeasure time until the page is ready for interaction consistently in both arms.

• MessageShow one short message about the existing campaign during loading.
• OfferKeep campaign terms, targets and other promotional areas unchanged.
• PresentationKeep the loading indicator and surrounding design unchanged.
• DiscoveryReview visits to promoted categories and products.
• VisibilityInterpret results against observed loading time without artificially extending it.

1. 1Post-load CTA clicks do not stand in for purchases. Judge the variant on purchase conversion rate across all eligible assigned visitors; looking only at visitors who waited long enough to see the message overstates its value.
2. 2If time to main content readiness grows or exits during loading rise in the variant, the message is adding to the wait, and a small lift in purchases is no reason to ship. Campaign visuals must not cover the loading indicator either; without it, visitors may think the app has frozen.
3. 3The finding is tied to this campaign and this message, so do not expect the same effect once the campaign changes. You can look at predefined contexts such as device or connection, but do not treat groups built afterward from loading time as if they were randomized arms.
4. 4A short message may move purchases too little for this traffic to detect when the gap is unclear. Compare the sample per variant that the sample size calculator gives for the current purchase conversion rate and the lift you care about with the visitors each arm actually received. If the test fell well short, do not read the result as proof of no effect.