Testing a One-Time Feature Tip Card on a Dashboard

This test makes sense when a valuable feature is available but seems undiscovered, such as accounts that never open it, support tickets asking for something the product already does, or survey answers from users who never heard of it. If the tip works, users who see it try the feature more often and come back to it after the first try. If people find the feature but it does not help them, a tip will not fix that, so work on the feature first; before launch, also confirm the card does not get in the way of the task the user is doing.

Feature trial rate

Among eligible users who have not used the feature before and were assigned in advance, compare first use within the defined period.

Also track

• Repeat feature useCompare repeat use over the same period and report the transition from first to repeat use separately.
• Time to complete the main taskMake sure the tip does not slow the task the user is trying to complete.
• Tip dismissal rateUse dismissal only as a diagnostic measure within the variant.
• Support contactsMake sure the tip does not increase confusion-related support requests.

• Single tipShow one predefined, dismissible card for a feature the user has not used.
• EligibilityUse actual usage history and a fixed display trigger.
• Keep constantKeep feature access, dashboard layout and the main task unchanged.
• FrequencyShow the card once; after dismissal, do not show it again during this test.
• Separate testsTest copy, timing, multiple tips and repeat display separately.

1. 1The feature trial rate is the primary measure, but if main-task time grows or support contacts rise, more first uses do not justify shipping the tip. That pattern may mean the tip gets in the way of the user's real work; read dismissal only as a diagnostic within the variant.
2. 2A first try does not show adoption on its own. Compare repeat feature use among all assigned users over the same period, report the move from first to repeat use separately, and treat the result as valid only for this feature and this tip.
3. 3Because only users who have never used the feature qualify, the eligible audience may be small. Work out how many users you need to detect the difference you expect with the sample size calculator, and if the result is unclear, check whether you reached that number before extending the test.