What to Put on a Zero-Results Search Page: A/B Test

Search logs make the case for this test when a noticeable share of queries return nothing and most of those visitors end their session on the empty screen; misspelled queries point the same way. When the recovery page does its job, visitors move on to a suggested category or product, and repeat searches and session continuation rise. If the real problem is that terms the catalog does carry fail to match, fix the matching first, start the test after that and leave the matching rules alone while it runs.

Zero-results recovery rate

Among eligible visitors assigned on the zero-results page, measure the share who reach and view a product page from search results or a relevant category within the follow-up window.

Also track

• Search again rateCompare new queries among visitors assigned to the zero-results page.
• Session continuation rateCompare visitors who continue interacting with the site in the same session window.
• Purchase after zero resultsCount purchases among all eligible assigned zero-result visitors during the follow-up period.
• Page load timeMeasure page speed after the recommendation block appears under the same device and network conditions.

• Recovery experienceAdd one predefined recovery experience combining spelling suggestions, relevant categories and suitable products.
• Content rulesKeep recommendation sources and eligibility fixed.
• Search fieldKeep the query editing field available and consistent in both arms.
• OrderKeep block order fixed; testing each block separately requires separate experiments.
• DeviceReview mobile and desktop separately.

1. 1Recovery rate counts visits to a product page; check whether those visits turn into sales with purchase after zero results, counted over an equal follow-up window. If page load time shows the blocks slowing the page down, the gain may exist only on paper.
2. 2Because the variant shows all three blocks together, the result belongs to the recovery page as a whole, and this test cannot tell you which block did the work. Split the read by device as well, and leave block-by-block comparisons to later tests.
3. 3Check that recommended products actually relate to the query and that the page has not turned into a campaign showcase; irrelevant suggestions may draw clicks but they mislead visitors. Offer the spelling suggestion as a choice and never swap the visitor's query for it automatically.
4. 4Visitors who hit zero results may be a limited group, so the difference can stay unclear for a long time. Feed the current recovery rate and the minimum lift worth acting on into the sample size calculator to find the visitors needed per arm. Divide that by the zero-result traffic each arm gets per day to fix the run length before launch, and do not call the test early.