A/B Test Idea: Home Screen for Returning App Users

Long routes back to unfinished tasks show up in three places: returning visitors who leave without reopening their carts, session recordings of people searching again for the same product, and repeated product searches in search logs. If the continue block shortens that route, returning visitors get back to their carts or recent tasks faster, and revenue per visitor reflects it. Confirm first that the fixed signal identifying returning visitors is reliable and that the block will not openly show a name, address or other personal details; with a shaky signal, misclassification will erode its value.

Revenue per visitor (RPV)

Compare revenue per assigned visitor within predefined new and returning segments, then report the overall result.

Also track

• Homepage click-through rateMeasure interaction with the first block; do not classify an impression as unnoticed just because it was not clicked.
• Return-to-cart rateTrack eligible returning visitors who resume their carts.
• New-user conversionCheck whether misclassification or the fallback experience harms new-user conversion.
• Page load timeMake sure personalization does not harm LCP.

• Continue blockAdd the predefined continue block for returning visitors in the variant.
• New visitorsKeep the new-visitor homepage unchanged in both arms.
• ClassificationUse one fixed signal and classify visitors before assignment.
• Content scopeKeep other blocks and recommendation logic unchanged.
• FallbackDefine the default experience for unknown or misclassified visitors.

1. 1Read revenue per visitor within the predefined new and returning segments first, then overall. If revenue rises among returning visitors while new-user conversion falls, misclassification or a broken fallback experience may be eating the gain; in that case, do not ship.
2. 2Homepage clicks and the return-to-cart rate explain the result but do not decide it; do not label an impression as unnoticed just because nobody clicked. Compare page load time in both arms; if personalization bypasses caching and slows the page, the revenue gap reflects speed as well as the block.
3. 3What you learn applies to this classification signal and this continue block; if you change the segment definition mid-test, the start and end of the test no longer measure the same audience. Redesigning the new-user homepage is a separate experiment.
4. 4If the revenue gap is small or noisy, swings in order value may be hiding it, and returning visitors are only part of your traffic. The sample size calculator works on conversion rates, so use the returning segment's cart return rate to get a sample per variant and divide it by daily returning visitors per arm for a minimum run length. Revenue is noisier, so treat that as a floor and do not stop the test on an early look.