Showing First-Run Tooltips in a Mobile App A/B Test

A core feature that new users rarely open, support tickets asking where it lives and session recordings of users wandering between menus all point to a discovery gap that first-use tips could close. If the tips work, users who see one use the promoted feature for the first time within the set period and still reach their first meaningful action. If the feature is found but not used, the problem is value, not discovery; keep tips skippable and do not force a long, required sequence before users try the product.

Promoted feature adoption rate

Among eligible users who have not used the feature and are assigned to the tip experience, compare first use within a predefined period.

Also track

• First-session completion rateCheck whether users reach their first meaningful action.
• Tip skip rateReport skips within the variant; there is no tip to skip in the control.
• Seven-day retentionCompare retention among all assigned users.
• First-session exit rateMake sure the tutorial burden does not increase exits.

• TipsAdd one predefined set of interface tips.
• SequenceKeep tip count, order and copy fixed.
• TriggerDefine when each tip appears and keep the rule consistent.
• Later accessKeep the chosen re-access behavior fixed during the test.
• AudienceStart with eligible first-time users; returning after an update is a separate cohort.

1. 1Higher use of the promoted feature counts for little until you check retention. Compare seven-day retention across all assigned users, not only those who saw the tips; if the first-session exit rate rises, the tutorial burden may be pushing people away.
2. 2This finding is specific to this tip set, its order and its trigger, and to eligible first-time users; test users returning after an update as a separate cohort before extending it. Read the tip skip rate within the variant only, and confirm that screen readers can navigate the tip layer.
3. 3A rarely used feature can leave the gap small simply because the predefined period is too short to capture its use. Work out how many users you need to detect the difference you care about with the sample size calculator; if you fall short, plan a new round instead of stretching the period mid-test.