Relevance vs Best-Selling Default Search Sort A/B Test

It is time to compare the two defaults when visitors on search results often switch away from the default sort, or skip the top products and scroll down, and when certain queries pile up searches with no interaction. The better default draws more clicks on the first four results and leaves fewer visitors changing the sort. If the problem lies in the result set itself, with irrelevant products showing up for a query, fix the matching first, because no ranking method will solve that.

Search purchase conversion rate (CR)

Compare purchases within the same follow-up window among eligible visitors who see search results; do not select the primary analysis based on post-assignment search behavior.

Also track

• Clicks on the first four resultsMeasure clicks on the first four product links using the same search event denominator.
• Sort changesCompare visitors who switch from the default sort to another option.
• Average order valueReport order value using the same discount and cancellation rules; track gross margin separately.
• Searches with no interactionCount queries where results were shown but no product click or other predefined search action occurred.

• Search conversionCompare relevance-based default ranking with best-selling default ranking for the same result set.
• Top resultsCompare clicks on the first four results in each arm.
• Sort changesSee which default visitors change more often.
• Order valueReview order value and gross margin alongside conversion.
• SegmentsReview new and returning users separately.

1. 1Even if one arm shows higher search purchase conversion, do not decide without checking average order value and gross margin, since best-seller ranking can shift sales toward different products. Clicks on the first four results and searches with no interaction explain the gap but do not pick the winner.
2. 2Run the main analysis on visitors who reach search results and were assigned before they saw either ranking; if you later narrow it to clickers or buyers, the arms stop being comparable. Read new and returning users separately, since each group may lean toward a different default. Before carrying the result over to category page sorting, test it there separately.
3. 3Before reading the result, confirm that sort options stayed visible in both arms, the filter set did not change and out-of-stock products did not rise to the top. If sponsored products were not clearly marked apart from organic results, the gap cannot be credited to the ranking method.
4. 4If the gap is small, enter each arm's visitors who saw search results and those who purchased into the A/B test significance calculator, and it will tell you whether the search purchase conversion gap goes beyond chance. Differences in average order value and margin are outside that calculation, so read them with the same caution.