Ali Demirbaş - Lab
TRGet in touch

Forms & Signup

One-Time Code vs Password Login A/B Test Example

A one-time code avoids remembering a password but adds a wait and often a switch to email or SMS.

  • Forms & signup
  • Sign-in completion rate

Control vs Variant

Two alternatives; neither is the incumbent.

Changed
Options
Page
Forms & signup
Difference
changed

Hypothesis

Delivery delays may interrupt sign-in. Choose one delivery channel before the test; do not turn this into an email-versus-SMS comparison.

How to run this test

Primary metric

Sign-in completion rate

Compare whether each method helps visitors complete sign-in.

Other metrics

  • Code delivery and useReport code delivery and use separately within the variant; the password arm has no equivalent events.
  • First-attempt success rateCheck whether visitors can sign in successfully on the first try.
  • Password reset requestsTrack whether reset requests decrease.
  • Support contactsMake sure sign-in-related support requests do not increase.

What to check before and during the test

  • MethodCompare password sign-in with a flow that starts with a code sent through one predefined channel.
  • AlternativeKeep password sign-in available in the variant.
  • Keep constantKeep security steps, code validity and delivery settings unchanged.
  • DeliveryReview delivery delays and whether users return from the delivery channel.
  • DeviceReview code autofill by device; do not change it during the test.

Setup mistakes to avoid

  1. 1Do not include two-step verification or identity verification in this test; these are security controls.
  2. 2Do not start while code delivery is unstable; that would test the delivery infrastructure.
  3. 3Do not remove password sign-in and leave no reversible alternative.
  4. 4Do not change code expiry during the test.
  5. 5Do not generalize results from new users to the existing user base.