Raising the Free Shipping Threshold A/B Test

Orders clustering around the threshold in the order value distribution, or shoppers leaving the cart once they see a shipping fee, show that the threshold is shaping decisions. If the new threshold works, shoppers close to it add items to clear it, and those who abandon do not cancel out that gain. If exits stem from something other than the shipping fee, such as payment errors, look into that before you change the threshold. Choose the alternative from the current basket distribution and shipping costs, and check before launch that the free-shipping message states its conditions accurately.

Revenue per visitor (RPV)

Compare net revenue per assigned visitor across the two shipping thresholds, accounting for order conversion and shipping cost.

Also track

• Average order value (AOV)Compare revenue per completed order; interpret it alongside conversion and RPV.
• Order completion rateCompare completed orders among all visitors assigned to each threshold.
• Cart-to-checkout rateCompare assigned cart visitors who start checkout.
• Shipping cost per orderTrack shipping expense and its effect on contribution margin.

• ThresholdCompare the current free-shipping threshold with one alternative chosen from unit economics and shipping costs; set both values before launch.
• Keep constantKeep prices, promotions, shipping message and its presentation unchanged.
• RevenueUse RPV to assess whether any increase in AOV offsets a change in conversion.
• CostInclude shipping expense and margin impact in the decision.
• AudienceReview new and returning visitors with the same definitions; keep each visitor in the same variant.

1. 1A higher AOV on its own is not a win. Decide on RPV and read the effect of shipping cost per order on margin with it; even with a higher RPV, do not switch to the new threshold if shipping cost is eating into margin.
2. 2What you learn describes only the two thresholds you compared; do not carry it over to another threshold value or a different promotional period. Split new and returning visitors with the same definitions, and check whether the cart-to-checkout rate differs between the arms.
3. 3Shoppers who add items and shoppers who abandon can offset each other, so the net difference may be small. Before launch, get the visitors needed per arm for the order completion rate from the sample size calculator; RPV is an average and noisier, so treat that number as a floor. If the difference is still unclear, do not conclude that the two thresholds perform the same.