Ali Demirbaş - Lab

UI Elements

Site-Wide Campaign Announcement Bar A/B Test Example

A temporary site-wide banner can expose the same campaign across more page visits.

  • UI elements
  • Campaign action completion rate

Control vs Variant

Absent on one side, present on the other.

Changed
Presence
Page
UI elements
Difference
added

Hypothesis

It takes up space and may attract less attention on repeat visits. Keep assignment consistent across the site and compare the presence of one fixed banner during the actual campaign period.

How to run this test

Primary metric

Campaign action completion rate

Choose one campaign action before launch, such as visiting the campaign page or completing an eligible campaign order, and measure it among all assigned visitors. Do not run the test until the action event is defined.

Other metrics

  • Overall purchase conversion rate (CR)Compare purchases or the predefined primary site action among all eligible assigned visitors.
  • Announcement bar click-through rateCompare clicks among eligible visitors who see the bar.
  • Bar dismissal rateMeasure dismissals; a dismissal alone does not prove irritation.
  • Page exit rateCompare exits using the same session window.

What to check before and during the test

  • PresenceAdd one fixed announcement bar to the predefined site-wide page scope.
  • CoverageKeep page coverage the same; testing selected categories is a separate experiment.
  • DismissalUse the same close behavior and respect dismissals on later pages.
  • Repeated viewsUse visitor-level exposure history rather than counting every page view as a separate person.
  • MobileCheck space taken and access to page content.

Setup mistakes to avoid

  1. 1Keep the announcement bar dismissible.
  2. 2Change presence only; keep its message unchanged.
  3. 3Remove the time-limited message when the campaign ends and use accurate dates.
  4. 4Do not let a fixed bar permanently cover important content.
  5. 5Do not call a winner from bar clicks alone; review the selected campaign action.