A/B testing examples: Real cases from Nike, Amazon and Etsy

Famous A/B testing examples are controlled experiments that well-known brands ran on their websites and apps and then published with results. This post covers nine real cases from Nike, Amazon, YSL Beauty, HP, Ubisoft, Discovery, Duolingo, Etsy and Durex. Each one comes either from the person or company that ran the test or from the testing vendor's official customer page. Every number is reported as the source states it.

Ad tests are left out: every case here is a change to a page or a screen. Alongside the winners you will see tests that disappointed, because that is where A/B testing teaches the most. Each case ends with a link to the matching test in the A/B test library. Use it if you want to try the same idea on your own site.

Most famous A/B testing examples fall into two groups. Some are small changes that made a bigger difference than expected. Others are ideas everyone was sure of that lost. The table summarizes the nine cases in this post: what was tested and the result.

Most of the lifts in the table are relative, meaning a percentage change from the starting rate. The Ubisoft case gives both the starting and the final rate, while most of the others give only the percentage. The last section of this post explains why that difference matters when you read a result.

Most published A/B test examples from ecommerce cluster in the cart, checkout and navigation steps. This is because a small point of friction in these steps shows up directly in orders, and its effect is easy to measure.

Nike: a sticky checkout button in the mobile cart

Will Close, Nike's Director of Digital Experiments, presented Nike.com tests at Opticon, the conference of the testing vendor Optimizely. Optimizely published the deck in 2019. According to the talk, the tests were run on Nike.com with a custom in-house testing tool by a team of four.

In the first case the problem was in the mobile cart: the primary button, "Checkout", sat out of view. The variant moved the button to the top and made it sticky while scrolling. The share of visitors going to checkout rose 5.5%, and conversion on multi-item orders rose 5.67%.

The second case concerns the membership pitch in checkout. The talk's diagnosis was that the membership value proposition was lengthy marketing-speak. Most people in checkout, it noted, want to place their order quickly rather than read. When the copy was cut to a simple consumer value proposition, selection of the membership checkbox rose 172% and new Nike members rose 30.3%.

The deck gives only relative lifts. It states no baseline rate, sample size, duration or confidence level. To try the same idea on your site, see the sticky buy button test.

Amazon: a credit card offer and cart recommendations

In 2004 Amazon's home page was split into slots, and the content for each slot was tested automatically. Amazon's credit card offer kept winning the top slot. Its clickthrough rate was very low, but the offer was so profitable that its expected value came out high.

The team decided the home page was not the right place for it. They moved the offer to the shopping cart page shown after an item is added. Next to it was simple math highlighting the savings relative to the items in the cart.

According to a 2014 paper by Ronny Kohavi and co-authors, the controlled experiment measured this change. It showed the simple change was worth tens of millions of dollars in profit annually. The source gives no more precise figure.

An older Amazon story is about shopping cart recommendations. Greg Linden, who worked at Amazon, writes about it on his blog. A marketing senior vice-president was against recommending items based on what was in the cart. The VP argued it might distract people from checking out.

Linden was told to stop working on it and prepared the feature for a test anyway. In his telling, the feature won by such a wide margin that not having it live was costing Amazon a noticeable amount of money. The post gives no percentage and no date.

The point of the case is that when an executive's intuition and a measured result disagree, a test can settle it. For a similar test, see product recommendations inside the cart.

YSL Beauty: the order of menu tabs

YSL Beauty, part of L'Oréal, was not sure what to emphasize in its home page menu: new products or offers. The testing vendor's consultants proposed swapping two tabs. In the variant, the "Exclusive Offers" tab, which leads to current offers and gifts with purchase, came before the "New Products" tab.

According to Wingify's case page, the test ran for a month on more than 150,000 visitors. Visits to the basket page rose 9%, visits to the checkout page rose 13%, and overall transactions rose 10%. The team sent all traffic to the winning variant and hardcoded the change into the site.

Wingify is the company formed when VWO and AB Tasty merged, and the figures in this case are reported by the vendor itself. For a test on menu order, see the menu category order test.

HP: a category page and a subscription offer

Optimizely's HP case page describes two concrete tests. In the first, HP's ecommerce UX team redesigned the printer category page. They ran a 50/50 split test of the old page against the new one. The new page increased success metrics by 20% and yielded an incremental $260K per quarter.

The second test concerns the enrollment offer for Instant Ink, HP's ink subscription service. The variant showed a free trial option and positioned the service as a printer feature. Enrollment rose 37%.

Two caveats apply. The page does not define which metrics "success metrics" covers. And because the whole category page was redesigned, the test shows that the new page is better but not which change made the difference. Whole-page tests speed up a decision, while single-variable tests teach you what works.

Ubisoft: a buy page without scrolling

Ubisoft wanted to simplify buying on the "Buy Now" page for its game For Honor. On the old page, visitors chose an edition, scrolled down to choose a platform, and finally clicked "Place Your Order".

Working from heatmaps, scroll maps and on-page surveys, the team formed a hypothesis: reduce scrolling up and down. The variant moved the edition, console and order step to the top of the left column, together with an edition comparison. The page needed no scrolling.

According to Wingify's case page, the test ran from October to December 2017, during the holiday season when traffic was 60% higher than usual. The new design raised conversion from 38% to 50%.

The page summarizes this as a "12% increase", but that is a 12-point absolute difference. In relative terms the lift is about 32%. Making this distinction when you read a case keeps you from mistranslating the number into your own expectations. For tests that cut scrolling and steps, see the one-page vs multi-step checkout test.

A/B testing is not limited to stores. A video site's content pages and a language app's most important feature were tested the same way.

Discovery: a slow panning effect on video stills

The team of Jeffrey Douglas, Director of Product at Discovery Digital Networks, measured engagement with the site's video content in clicks and video views. Douglas set up what he called the "Ken Burns Test". It is named after the documentary filmmaker who brings still photos to life by slowly panning the camera across them. The team tested a similar panning effect against the original static video stills on its video content pages.

According to Optimizely's case page, the test ran on approximately 20,000 visitors to Discovery's website. The effect increased click-throughs into the video content by 6%. The team then applied the learning to videos across the site. The page gives no date or confidence level.

Depending on the baseline rate, about 20,000 visitors can be too few to reliably detect a relative difference as small as 6%. Read this result as an idea worth testing rather than a firm rule. To compare motion and still imagery in a hero area, see the autoplaying hero video vs static image test.

Duolingo: separating the streak from the daily goal

Every Duolingo learner has a daily goal, a set number of lessons to complete each day. According to Duolingo's November 2020 post, learners could until then only extend their streak when they reached that goal. The data showed that learners with higher daily goals were less likely to be on a streak, even when they used the app every day. Almost 40% of learners active two days in a row with no streak had chosen the highest goal tier.

The team tested separating the two mechanics. In the variant, a single lesson extended the streak and progress toward the daily goal was shown separately.

The post says Duolingo A/B tests nearly every change it makes, and all results are reported as relative changes. Day 14 retention rose 3.3% and daily active learners rose 1%. Within 20 days, the share of daily learners on a streak rose 10.5%.

The post is open about the downside: without the streak as motivation, fewer learners reached their daily goals. Because more people were building a daily habit overall, the team kept the change. The case shows that a test is judged by weighing metrics that move in opposite directions, not by a single number.

A/B tests that lose show that an assumption the team was sure of is wrong, and they show it before a permanent change ships. Most published cases describe winning tests, but the two cases below show why an idea that looks good should be tested first.

Etsy: infinite scroll made search worse

Dan McKinley, who worked at Etsy, described the company's experiment with infinite scroll in search results in his December 2012 talk "Design for Continuous Experimentation". The expectation was that people would see more items and buy more. The result was the opposite.

The median number of items seen was 40 for visitors with infinite scroll and 80 for the control group. People in the variant clicked fewer results, saved fewer items as favorites and bought fewer items from search. They did not buy less overall, they just stopped using search to find items. In McKinley's words, they had made search worse.

The team then tested the two assumptions the idea rested on separately. Showing more results per page barely, if at all, increased visits to item pages and made absolutely no change in purchases. Artificially slowing search down changed nothing either.

The lesson is to check the assumptions behind a big feature with cheap tests before you build it. To ask the same question on your own listing pages, see the infinite scroll vs load more button test.

Durex: ideas copied from other brands

Optimizely's case page is open about ideas Reckitt tried on the Durex UK site that did not work. The team first tried a Kylie Cosmetics style navigation that showcased best sellers, and revenue fell 6%. The team concluded that Durex customers preferred to shop by category rather than by best sellers. A desktop burger menu similar to Kylie Skin's cut revenue by 7% over a two-week test.

The third attempt was an Amazon-style bundle offer. Customers who had selected one product would pay a £4.99 postage fee. They were offered a bundle of related products that took them over the free delivery threshold. In the test, 1% fewer people added anything to their cart, average order value was 1% lower, and total revenue fell 6%.

The same page also describes a winner. Using short lines instead of long brand headlines increased product page visits by 32% and revenue by 6% in one month. To test a bundle offer on your own site, see the bundle options on the product page test.

The main takeaway from these cases is that it is hard to know in advance whether a change will work. That is why it needs to be tested. The three questions below are the ones that will help most when you use these cases for your own decisions.

What share of A/B tests actually win?

Only a small share of A/B tests win. According to a Harvard Business Review article, only about 10% to 20% of experiments at Google and Bing generate positive results. Its authors are Ronny Kohavi, who led Microsoft's experimentation team, and Stefan Thomke of Harvard. At Microsoft as a whole, one-third prove effective, one-third have neutral results and one-third have negative results.

The cases in this post do not reflect that rate, because companies and testing vendors mostly publish winners. When you read a list of cases, it is more accurate to think "this idea worked once somewhere" than "this idea works everywhere".

Can you copy another company's test results?

You cannot copy another company's test result, but you can use it as a hypothesis to test. The Durex case shows this well: navigation and bundle ideas that worked for other brands lowered revenue with a different audience.

When you move a result to your own site, the audience, product, price, traffic sources and the rest of the page all change. Nike's sticky checkout button or Amazon's cart recommendations are worth testing on your site, but only your own test will tell you whether they work. To plan the hypothesis, metric and setup, see the complete A/B testing guide.

Can you trust the numbers in A/B test case studies?

You can trust case study numbers as far as the source supports them. Most sources leave out some of what you would need to verify a result. Check four things when you read a case:

• Relative or absolute? Ubisoft's "12%" is actually a 12-point difference, about 32% in relative terms. Do not translate the number into your own expectations before you know which one is meant.
• When was it measured? The Ubisoft test ran during the holiday season, when traffic was 60% higher than usual. Seasonal visitors may behave differently from those in the rest of the year.
• Are sample size and significance given? The Nike, HP and Durex cases give no sample size or confidence level. The YSL and Discovery cases give only a visitor count.
• Who is telling the story? Testing vendors write their case pages to promote their products, and they usually pick tests that succeeded.

Ask the same questions when you read your own results. The guide to statistical significance in A/B testing explains whether a difference can be put down to chance. The sample size guide shows how many visitors a test needs. If you are looking for ideas to test on your own site, the A/B test library organizes scenarios by page type and goal.