Ali Demirbaş - Lab
TRGet in touch

Mobile App

Showing First-Run Tooltips in a Mobile App A/B Test

Brief tips may introduce features new users might otherwise miss.

  • Mobile app
  • Promoted feature adoption rate

Control vs Variant

Absent on one side, present on the other.

Changed
Presence
Page
Mobile app
Difference
added

Hypothesis

Too many interruptions, or tips shown at the wrong time, may make it harder to try the product. Measure actual use of the introduced features, not completion of the tip sequence.

How to run this test

Primary metric

Promoted feature adoption rate

Among eligible users who have not used the feature and are assigned to the tip experience, compare first use within a predefined period.

Other metrics

  • First-session completion rateCheck whether users reach their first meaningful action.
  • Tip skip rateReport skips within the variant; there is no tip to skip in the control.
  • Seven-day retentionCompare retention among all assigned users.
  • First-session exit rateMake sure the tutorial burden does not increase exits.

What to check before and during the test

  • TipsAdd one predefined set of interface tips.
  • SequenceKeep tip count, order and copy fixed.
  • TriggerDefine when each tip appears and keep the rule consistent.
  • Later accessKeep the chosen re-access behavior fixed during the test.
  • AudienceStart with eligible first-time users; returning after an update is a separate cohort.

Setup mistakes to avoid

  1. 1Do not make tips impossible to skip.
  2. 2Do not change tip count and wording together.
  3. 3Do not require users to complete a long sequence before trying the product.
  4. 4Make sure screen readers can navigate the tip layer.
  5. 5Do not call a win from feature use without checking retention.