Site-Wide Campaign Announcement Bar A/B Test Example

A time-limited campaign announced only on the home page or the campaign page can leave visitors elsewhere unaware of it; few visits to the campaign page, site searches for the campaign or support tickets from customers who heard about it too late are the warning signs. When the banner does its job, visitors who see it on different pages complete the predefined campaign action more often. Before launch, choose that action, confirm the campaign is real with accurate dates and countdown, and make sure the banner comes down when it ends.

Campaign action completion rate

Choose one campaign action before launch, such as visiting the campaign page or completing an eligible campaign order, and measure it among all assigned visitors. Do not run the test until the action event is defined.

Also track

• Overall purchase conversion rate (CR)Compare purchases or the predefined primary site action among all eligible assigned visitors.
• Announcement bar click-through rateCompare clicks among eligible visitors who see the bar.
• Bar dismissal rateMeasure dismissals; a dismissal alone does not prove irritation.
• Page exit rateCompare exits using the same session window.

• PresenceAdd one fixed announcement bar to the predefined site-wide page scope.
• CoverageKeep page coverage the same; testing selected categories is a separate experiment.
• DismissalUse the same close behavior and respect dismissals on later pages.
• Repeated viewsUse visitor-level exposure history rather than counting every page view as a separate person.
• MobileCheck space taken and access to page content.

1. 1A higher announcement bar click-through rate does not settle the test. The predefined campaign action completion rate is the deciding metric; check that overall purchase conversion and page exit rate do not worsen over the same window. Do not read the bar dismissal rate as proof of irritation on its own.
2. 2This test measures only whether the banner is present; if you also change its message, you cannot tell what caused the difference. Read the result by each visitor's exposure history rather than counting every page view as a separate person, and check on mobile that the fixed banner can be dismissed and never permanently covers important content.
3. 3If the result is not clear-cut, first check that assignment stayed consistent across the site and that visitors split between the arms as planned (sample ratio mismatch). Because the campaign window is limited, use the sample size calculator before launch to find the visitors per arm needed for the effect you care about, and check that eligible traffic per arm during the campaign can reach it.