# One-Time Code vs Password Login A/B Test Example

Page: https://alidemirbas.com.tr/en/lab/ab-test-library/tests/one-time-code-vs-password-login-ab-test-example

## Hypothesis

If sign-in starts with a one-time code sent through the predefined channel instead of a password, users who forget their password can get in without falling into the reset flow. Sign-in completion rate could go up, although waiting for the code and switching to the channel and back may cancel part of the gain.

## When to run this test

Frequent password reset requests, repeated failed attempts on the sign-in screen, or support contacts about forgotten passwords point to the password as what stops people at sign-in. If codes work better, users receive the code, get in on the first try and file fewer reset requests. If code delivery is unstable, fix known delivery failures and tracking errors first; otherwise you end up measuring the delivery infrastructure instead of the sign-in method.

## Primary metric

**Sign-in completion rate**: Compare whether each method helps visitors complete sign-in.

## Other metrics

- **Code delivery and use**: Report code delivery and use separately within the variant; the password arm has no equivalent events.
- **First-attempt success rate**: Check whether visitors can sign in successfully on the first try.
- **Password reset requests**: Track whether reset requests decrease.
- **Support contacts**: Make sure sign-in-related support requests do not increase.

## What to check before and during the test

- **Method**: Compare password sign-in with a flow that starts with a code sent through one predefined channel.
- **Alternative**: Keep password sign-in available in the variant.
- **Keep constant**: Keep security steps, code validity and delivery settings unchanged.
- **Delivery**: Review delivery delays and whether users return from the delivery channel.
- **Device**: Review code autofill by device; do not change it during the test.

## What to watch when reading the result

- Base the decision on the sign-in completion rate, with first-attempt success rate as supporting evidence. If password reset requests fall while sign-in-related support contacts rise, the load may simply have moved, so do not count that as a gain.
- Do not compare code delivery and use across arms, because the password arm has no matching events; report them within the variant as diagnostics. Normal delivery delay is part of the real experience, so attribute the result to the flow you tested, not to an ideal code method.
- This result is tied to the chosen delivery channel and the users you tested. If you measured only new users, do not extend it to your existing user base, whose password habits may differ. Code autofill varies by device, so read results by device as well.
- Do not try to close a small gap by loosening two-step verification or identity verification; they are security controls and sit outside this test. Before you decide, check with the [A/B test significance calculator](https://alidemirbas.com.tr/en/calculators/ab-test) whether the gap in sign-in completion is statistically significant.

## Frequently Asked Questions

### Should the code be delivered by email or SMS?

Choose using user access, delivery reliability, delay, cost and security requirements. Hold the selected channel fixed in this comparison. Email versus SMS needs a separate delivery-channel experiment.

### Can delivery and redemption rates be compared across both arms?

Not directly: the password arm has no equivalent code events. Report them within the variant. Use login completion as the primary outcome, with first-attempt success as supporting evidence.

### Should the test start while delivery is unstable?

Fix known delivery failures and instrumentation errors first. Normal delivery delay remains part of the real login experience. Monitor it rather than attributing results to an idealized code method.

### Can the result be generalized to all users?

Not if only new users were tested. Existing password habits may differ. Inspect code autofill by device and monitor recovery and login-related support.