A/B Test Significance Calculator
Enter visitors and conversions for control and variant to get the p-value, the z-score and whether the A/B test result is significant at 95%.
p-value
0.0768
Enter your numbers and press Calculate.
z-score = (Variant rate − Control rate) ÷ Standard error
How to calculate statistical significance for an A/B test
z = (p₂ − p₁) ÷ √[p̂ × (1 − p̂) × (1/n₁ + 1/n₂)], where p̂ = (x₁ + x₂) ÷ (n₁ + n₂) and p-value = 2 × (1 − Φ(|z|))
- Work out each conversion rate: p₁ = control conversions ÷ control visitors and p₂ = variant conversions ÷ variant visitors.
- Pool both groups into one rate, p̂ = all conversions ÷ all visitors, and use it to get the standard error of the difference.
- Divide the difference in rates by the standard error to get z, then read the two-sided p-value from the standard normal distribution. Below 0.05 counts as significant at 95% confidence.
250 ÷ 5,000 = 5.00% vs 290 ÷ 5,000 = 5.80%, pooled rate 5.40%, standard error ≈ 0.00452, z ≈ 1.77, p ≈ 0.0768 (not significant at 95%)
The guide to this topic: Statistical significance in A/B testing: p-values explained
What is A/B test statistical significance
Statistical significance in an A/B test tells you whether the gap between control and variant is larger than random assignment alone would plausibly produce. This A/B test calculator answers that with a two-proportion z-test. It compares the two conversion rates, measures the gap in units of standard error (the z-score) and turns that into a p-value. If the p-value is below 0.05, the result is significant at the 95% confidence level the tool uses.
The test starts from a null hypothesis: both versions have the same true conversion rate and the observed difference is noise. The p-value is the probability of seeing a difference at least as large as yours if that null hypothesis were true. It is a statement about the data under an assumption. It does not give the probability that the variant is better, and it says nothing about how big the effect is.
The method follows the two-sample test for proportions in the NIST/SEMATECH e-Handbook of Statistical Methods. It pools both samples to estimate the shared rate under the null hypothesis and relies on the normal approximation to the binomial. That approximation needs reasonably large samples. Treat a result built on fewer than about 30 conversions per variant as low-confidence, even when the p-value clears 0.05. For the wider testing process around this number, from hypothesis to rollout, see the complete guide to A/B testing.
How sample size changes the p-value
The same lift can be significant or not depending on how much traffic stands behind it. The table keeps the example's rates fixed (5.0% for control, 5.8% for variant, a 16% relative lift) and changes only the number of visitors per variant. Every row is calculated with the formula above.
| Visitors per variant | Conversions (control / variant) | z-score | p-value | Significant at 95% |
|---|---|---|---|---|
| 2,500 | 125 / 145 | 1.25 | 0.211 | No |
| 5,000 | 250 / 290 | 1.77 | 0.0768 | No |
| 10,000 | 500 / 580 | 2.50 | 0.0123 | Yes |
| 20,000 | 1,000 / 1,160 | 3.54 | 0.0004 | Yes |
Nothing about the variant changed between rows. Only the evidence grew. That is why a non-significant result on a small test is weak evidence of no effect. It is also why you should fix the visitor count before launch with the A/B test sample size calculator. It works the other way too: a large enough sample can make a tiny, commercially irrelevant lift significant. Read the absolute and relative uplift next to the p-value.
This is the heart of statistical vs practical significance. Statistical significance asks whether the gap is bigger than chance. Practical significance asks whether the gap is worth shipping once build cost, maintenance and risk are counted. Check the second by turning the lift into money. A 0.8-point gain in conversion rate means far more revenue for a store with a high average order value than for one with a low one.
Before you trust a significant result
A p-value below 0.05 is only as reliable as the test that produced it. Go through these checks before you call a winner:
Planned sample size: stopping the first time the p-value dips under 0.05 inflates false positives. Evan Miller shows that repeated significance testing always increases the rate of false positives. He also shows that peeking at an experiment ten times turns what looks like 1% significance into 5% (How Not To Run an A/B Test).
Traffic split: if a 50/50 test ends with visibly uneven groups, check the assignment and tracking before reading the p-value. A broken split can create a significant-looking gap that has nothing to do with the change.
Number of comparisons: testing more than two variants, or several metrics, gives chance more opportunities to produce a false winner. Compare each variant with control and tighten the threshold. The Bonferroni method in the NIST e-Handbook splits the error rate across the number of comparisons. With three variants against control, each would need p below about 0.0167 instead of 0.05.
Full weeks: weekday and weekend visitors often behave differently, so a test that ran Monday to Thursday describes only part of your audience.
Enough conversions for the normal approximation: with very few conversions per variant, treat even a significant result as provisional.
A result that passes all five checks is a reasonable basis for a decision. A Bayesian analysis answers a related question, the probability that the variant beats control, and near the threshold it can disagree with this frequentist z-test. Pick one method before the test starts and stay with it.
When to use the significance calculator
Use the A/B test significance calculator once, after the test has reached the sample size you planned before launch. It fits any comparison of two conversion rates measured on randomly split groups: two landing pages, two checkout flows, two email subject lines. Plan the traffic first with the sample size calculator, then come back here with the final counts.
Where a significance test misleads
Checking the p-value every day and stopping at the first value under 0.05 makes the stated 5% error rate meaningless. A non-significant result can mean there is no real difference, or that the test was too small to detect the one that exists. The calculator also applies a fixed two-sided 95% threshold, so a result at p = 0.06 is reported as not significant even though the evidence sits close to the line.
Guides on this topic
Frequently Asked Questions
How do you calculate statistical significance for an A/B test?
Divide conversions by visitors for each group to get two conversion rates. Pool both groups to get the shared rate and use it to compute the standard error of the difference. Then divide the gap between the two rates by that standard error to get the z-score. The two-sided p-value comes from the standard normal distribution. With 250 conversions from 5,000 control visitors and 290 from 5,000 variant visitors, z is about 1.77 and p about 0.0768. That is above 0.05, so the result is not significant at 95%.
What does the p-value mean in an A/B test?
The p-value is the probability of seeing a difference at least as large as the observed one if control and variant had exactly the same true conversion rate. A p-value of 0.0768 means that, with no real difference, random assignment alone would produce a gap this large or larger about 7.7% of the time. It is not the probability that the null hypothesis is true, and it is not the probability that the variant is better.
Does 95% confidence mean there's a 95% chance the variant is better?
No. A result significant at 95% confidence means a difference this large would appear by chance less than 5% of the time if the two versions truly performed the same. The probability that the variant beats control is a different quantity, and it needs a Bayesian calculation with a prior. The two approaches answer related questions in different ways and can give different-sounding answers for the same data. When you report a result, state the p-value and the observed lift instead of translating 95% into a win probability.
My test isn't significant. Does that mean there's no difference?
Not necessarily. A non-significant result fits two situations: there is no meaningful difference, or the test was too small to detect the one that exists. The p-value alone cannot tell them apart. Enter your baseline rate and the lift you hoped to detect into the sample size calculator. If the required sample is much larger than what the test collected, the test was underpowered and the honest conclusion is "inconclusive". The fix is a longer or larger test. Re-slicing the same data until a segment crosses 0.05 only adds false positives.
Can I stop an A/B test as soon as it becomes significant?
Not with a fixed-sample test. Stopping at the first moment the p-value drops under 0.05 pushes the false-positive rate well above 5%. The p-value moves up and down as data arrives, and each look is another chance for noise to cross the line. Evan Miller's analysis of repeated significance testing shows that peeking at an experiment ten times turns what looks like 1% significance into 5% (How Not To Run an A/B Test). Fix the sample size before launch, run to it and check significance once. If you need to monitor early, use a sequential method built for that.
Related calculators