Ali Demirbaş - Lab
TRGet in touch

Search & Filtering

Relevance vs Best-Selling Default Search Sort A/B Test

Relevance ranking prioritizes match to the query; best-selling ranking prioritizes popularity among eligible results.

  • Search
  • Search purchase conversion rate (CR)

Option A vs Option B

The sequence is the variable.

Changed
Order
Page
Search
Difference
changed

Hypothesis

Keep the product pool for each query fixed so the comparison tests ranking alone. Preserve users’ ability to change the sort in both arms.

How to run this test

Primary metric

Search purchase conversion rate (CR)

Compare purchases within the same follow-up window among eligible visitors who see search results; do not select the primary analysis based on post-assignment search behavior.

Other metrics

  • Clicks on the first four resultsMeasure clicks on the first four product links using the same search event denominator.
  • Sort changesCompare visitors who switch from the default sort to another option.
  • Average order valueReport order value using the same discount and cancellation rules; track gross margin separately.
  • Searches with no interactionCount queries where results were shown but no product click or other predefined search action occurred.

What to check before and during the test

  • Search conversionCompare relevance-based default ranking with best-selling default ranking for the same result set.
  • Top resultsCompare clicks on the first four results in each arm.
  • Sort changesSee which default visitors change more often.
  • Order valueReview order value and gross margin alongside conversion.
  • SegmentsReview new and returning users separately.

Setup mistakes to avoid

  1. 1Do not hide sort options during the test.
  2. 2Do not rank out-of-stock products at the top.
  3. 3Change ranking only; keep filters the same.
  4. 4Do not present sponsored products as organic results.
  5. 5Review margin as well as conversion.