# Showing First-Run Tooltips in a Mobile App A/B Test

Page: https://alidemirbas.com.tr/en/lab/ab-test-library/tests/showing-first-run-tooltips-in-a-mobile-app-ab-test

## Hypothesis

If new users see short tooltips that point to hard-to-find features on first use, they may reach those features on their own. The promoted feature adoption rate could then rise among eligible users who have not used the feature before.

## When to run this test

A core feature that new users rarely open, support tickets asking where it lives and session recordings of users wandering between menus all point to a discovery gap that first-use tips could close. If the tips work, users who see one use the promoted feature for the first time within the set period and still reach their first meaningful action. If the feature is found but not used, the problem is value, not discovery; keep tips skippable and do not force a long, required sequence before users try the product.

## Primary metric

**Promoted feature adoption rate**: Among eligible users who have not used the feature and are assigned to the tip experience, compare first use within a predefined period.

## Other metrics

- **First-session completion rate**: Check whether users reach their first meaningful action.
- **Tip skip rate**: Report skips within the variant; there is no tip to skip in the control.
- **Seven-day retention**: Compare retention among all assigned users.
- **First-session exit rate**: Make sure the tutorial burden does not increase exits.

## What to check before and during the test

- **Tips**: Add one predefined set of interface tips.
- **Sequence**: Keep tip count, order and copy fixed.
- **Trigger**: Define when each tip appears and keep the rule consistent.
- **Later access**: Keep the chosen re-access behavior fixed during the test.
- **Audience**: Start with eligible first-time users; returning after an update is a separate cohort.

## What to watch when reading the result

- Higher use of the promoted feature counts for little until you check retention. Compare seven-day retention across all assigned users, not only those who saw the tips; if the first-session exit rate rises, the tutorial burden may be pushing people away.
- This finding is specific to this tip set, its order and its trigger, and to eligible first-time users; test users returning after an update as a separate cohort before extending it. Read the tip skip rate within the variant only, and confirm that screen readers can navigate the tip layer.
- A rarely used feature can leave the gap small simply because the predefined period is too short to capture its use. Work out how many users you need to detect the difference you care about with the [sample size calculator](https://alidemirbas.com.tr/en/calculators/sample-size-calculator); if you fall short, plan a new round instead of stretching the period mid-test.

## Frequently Asked Questions

### Why is completing the tip sequence not success?

The goal is using the product feature, not watching instructions. Use first meaningful feature use as the primary outcome. Sequence completion and skipping are variant-only diagnostics.

### Can tips prevent new users from trying the product?

Yes. Irrelevant or excessive interruptions can delay exploration. Keep tips skippable and monitor first-session exits and access to the first valuable action.

### Should first-use tips be shown only to new users?

This scenario targets first use. Existing users unfamiliar with a new feature need a separately defined layer or experiment rather than an unannounced expansion of eligibility.

### Does more feature use settle the decision?

Use the predefined first-use outcome and retention guardrail across assigned users. Check accessibility of the tip layer rather than analyzing only exposed or engaged users.