Forms & Signup
One-Time Code vs Password Login A/B Test Example
A one-time code avoids remembering a password but adds a wait and often a switch to email or SMS. Delivery delays may interrupt sign-in. Choose one delivery channel before the test; do not turn this into an email-versus-SMS comparison.
A/B test scenario. This page is a test plan, not the result of a real experiment.
- Forms & signup
- Sign-in completion rate
Control vs Variant
The current version against one alternative.
- Changed
- Options
- Page
- Forms & signup
- Difference
- changed
Hypothesis
If sign-in starts with a one-time code sent through the predefined channel instead of a password, users who forget their password can get in without falling into the reset flow. Sign-in completion rate could go up, although waiting for the code and switching to the channel and back may cancel part of the gain.
Test plan
When to run this test
Frequent password reset requests, repeated failed attempts on the sign-in screen, or support contacts about forgotten passwords point to the password as what stops people at sign-in. If codes work better, users receive the code, get in on the first try and file fewer reset requests. If code delivery is unstable, fix known delivery failures and tracking errors first; otherwise you end up measuring the delivery infrastructure instead of the sign-in method.
How to measure success
Sign-in completion rate
Compare whether each method helps visitors complete sign-in.
Also track
- Code delivery and useReport code delivery and use separately within the variant; the password arm has no equivalent events.
- First-attempt success rateCheck whether visitors can sign in successfully on the first try.
- Password reset requestsTrack whether reset requests decrease.
- Support contactsMake sure sign-in-related support requests do not increase.
How to set it up
- MethodCompare password sign-in with a flow that starts with a code sent through one predefined channel.
- AlternativeKeep password sign-in available in the variant.
- Keep constantKeep security steps, code validity and delivery settings unchanged.
- DeliveryReview delivery delays and whether users return from the delivery channel.
- DeviceReview code autofill by device; do not change it during the test.
What to watch when reading the result
- 1Base the decision on the sign-in completion rate, with first-attempt success rate as supporting evidence. If password reset requests fall while sign-in-related support contacts rise, the load may simply have moved, so do not count that as a gain.
- 2Do not compare code delivery and use across arms, because the password arm has no matching events; report them within the variant as diagnostics. Normal delivery delay is part of the real experience, so attribute the result to the flow you tested, not to an ideal code method.
- 3This result is tied to the chosen delivery channel and the users you tested. If you measured only new users, do not extend it to your existing user base, whose password habits may differ. Code autofill varies by device, so read results by device as well.
- 4Do not try to close a small gap by loosening two-step verification or identity verification; they are security controls and sit outside this test. Before you decide, check with the A/B test significance calculator whether the gap in sign-in completion is statistically significant.
Frequently Asked Questions
Should the code be delivered by email or SMS?
Choose using user access, delivery reliability, delay, cost and security requirements. Hold the selected channel fixed in this comparison. Email versus SMS needs a separate delivery-channel experiment.
Can delivery and redemption rates be compared across both arms?
Not directly: the password arm has no equivalent code events. Report them within the variant. Use login completion as the primary outcome, with first-attempt success as supporting evidence.
Should the test start while delivery is unstable?
Fix known delivery failures and instrumentation errors first. Normal delivery delay remains part of the real login experience. Monitor it rather than attributing results to an idealized code method.
Can the result be generalized to all users?
Not if only new users were tested. Existing password habits may differ. Inspect code autofill by device and monitor recovery and login-related support.
More tests in Forms & Signup
All testsDelaying Signup Until After the Action: A/B Test
Forms & signupAction completion rate
Asking users to register before the action, or moving that step after it. Includes the intent-filter argument and the cost of unregistered completions.
Modal Form vs In-Page Form A/B Test Example
Forms & signupForm completion rate
The same form opened in a modal layer or placed inline in its own section. Read focus against lost context, back-button behavior and accidental close.
Should a Multi-Step Form Show a Back Button? A/B Test
Forms & signupForm completion rate
A step screen with a forward action only, against the same screen with a visible back button. Covers correction, control and losing entered data.