# Statistical significance in A/B testing: p-values explained

Page: https://alidemirbas.com.tr/en/blog/statistical-significance-ab-testing

2026-11-01 · A/B Testing · Ali Demirbaş

What statistical significance and p-values mean in A/B testing, what they don't, with a worked example, Type I and II errors and confidence intervals.

Statistical significance tells you how hard it would be to explain the difference you see in an A/B test by chance alone. The p-value is the number behind it. Neither one directly answers "is the variant better?", but together they limit the risk of a wrong decision.

This guide explains statistical significance with a worked example. It covers what a p-value does not mean, where the 95% standard comes from, and Type I and Type II errors and confidence intervals. It also covers the three mistakes that most often break an A/B test result. To run the numbers on your own data, use the [A/B test significance calculator](https://alidemirbas.com.tr/en/calculators/ab-test).

## What is statistical significance in A/B testing?

Statistical significance in A/B testing is about the difference between control and variant. It means that difference would rarely appear if the two versions actually performed the same. A test starts from a null hypothesis that says there is no real difference.

The less the data fits that assumption, the smaller the p-value. When the p-value falls below a threshold you chose in advance, the significance level or alpha, the result is called statistically significant. The most common threshold in A/B testing is 0.05, which corresponds to a 95% confidence level.

Significance is not certainty. Even when there is no real difference, tests run at the 0.05 threshold will sometimes come out significant. The threshold sets how often you accept those false alarms. The NIST statistics handbook defines the significance level the same way: the risk of rejecting the null hypothesis when it is in fact true.

### Worked example: same lift, two results

An example shows why significance depends on more than the size of the difference. In both scenarios below, control converts at 4.0% and the variant at 4.6%, a 15% relative lift. The only thing that changes is the number of visitors per group. Both were calculated with a two-sided z-test for the difference between two proportions.

*Hypothetical example, calculated with a two-sided z-test for the difference between two proportions.*

| Scenario | Visitors per group | Conversions (control / variant) | p-value | 95% CI for the difference | Result |
| --- | --- | --- | --- | --- | --- |
| A | 10,000 | 400 / 460 | 0.037 | +0.04 to +1.16 points | Significant |
| B | 2,000 | 80 / 92 | 0.35 | −0.66 to +1.86 points | Not significant |

The same 15% lift is significant with the larger sample and not with the smaller one. In scenario B the confidence interval includes zero. That means the data fits a variant that is slightly worse than control just as well as one that is clearly better. That is why you decide how many visitors you need before the test starts, with the [sample size calculator](https://alidemirbas.com.tr/en/calculators/sample-size-calculator).

## What does a p-value not mean?

A p-value measures how often a result at least as extreme as yours would appear if the null hypothesis were true. Most mistakes come from reading more into it than that.

In 2016 the American Statistical Association (ASA) published six principles against common misreadings, and the full text is in the [ASA statement on p-values](https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf). Three matter most for A/B testing. A p-value is not:

- **The probability that the hypothesis is true:** A p-value of 0.03 does not mean "a 97% chance the variant is better". The ASA's second principle says p-values do not measure the probability that the studied hypothesis is true. They also do not measure the probability that the data were produced by random chance alone.
- **The size or importance of the effect:** With a very large sample, a trivial difference can produce a tiny p-value. The fifth principle says a p-value does not measure the size of an effect or the importance of a result.
- **A decision rule on its own:** The third principle says conclusions and business decisions should not be based only on whether a p-value passes a specific threshold.

In practice, report the p-value together with the size of the difference, its confidence interval and the business metric. A decision maker needs the size of the effect. "p = 0.037, the variant won" tells them far less than "conversion rose 0.6 points, 95% CI 0.04 to 1.16 points, p = 0.037".

## How do you choose a significance level?

You choose a significance level by weighing what each kind of error would cost. The 95% standard is a convention that stuck.

It is usually traced to Ronald Fisher's 1925 book. There he wrote that taking P = .05, or "1 in 20", as a limit was "convenient". The passage is in [chapter three of the book](https://psychclassics.yorku.ca/Fisher/Methods/chap3.htm). The threshold has been widely used since, which does not make it right for every decision.

If shipping a false winner would be expensive, for example a pricing or checkout change, a stricter threshold such as 0.01 can make sense. For a low-risk copy change that is easy to roll back, a looser 0.10 may be acceptable. Whatever you pick, write it down before the test and do not change it after results come in.

### Should you use a one-tailed or two-tailed test?

A two-tailed test allows for the variant being either better or worse than control. A one-tailed test checks only one direction. On the same data, the one-tailed p-value is half the two-tailed one. In scenario A above, the two-tailed p-value is 0.037 and the one-tailed p-value is 0.018.

Tools make different choices. Firebase A/B Testing calculates p-values with a one-tailed test in which the variant is greater than the baseline. Optimizely says Stats Engine uses two-tailed tests because its false discovery rate control requires them.

Because a change can also hurt the metric, a two-tailed test is the safer default. Do not compare results from two tools without knowing which one each uses.

## What are Type I and Type II errors?

A Type I error is concluding there is a difference when there is none, a false positive. Its probability equals the significance level, alpha. A Type II error is missing a difference that exists, a false negative, with probability beta. A test's power is 1 − beta, the probability of detecting a real effect.

*Alpha and beta are chosen before the test starts, and power is 1 − beta.*

| Error | What happens | Probability | How to reduce it |
| --- | --- | --- | --- |
| Type I (false positive) | A variant that does not work gets shipped | Alpha, usually 0.05 | Stricter threshold, no peeking, fewer metrics |
| Type II (false negative) | A variant that works gets discarded | Beta, where power = 1 − beta | Larger sample, bolder change |

The two are linked. With the same sample, lowering alpha cuts Type I errors and raises Type II errors. The way to reduce both is a larger sample.

Power and the minimum detectable effect belong to [sample size planning](https://alidemirbas.com.tr/en/blog/calculate-ab-test-sample-size). Type II error is also why "the test wasn't significant, so the idea is bad" is so often the wrong conclusion.

### How do confidence intervals relate to p-values?

A confidence interval gives the range of values around the observed difference that are consistent with the data. The p-value answers one narrow question: can the difference be told apart from zero? The interval shows how uncertain you are about the size of the difference.

For the same two-sided test and assumptions, a 95% confidence interval that excludes zero corresponds to a p-value below 0.05. In scenario A the interval starts at +0.04 points, excludes zero, and the p-value is below 0.05. In scenario B the interval includes zero and the p-value is above 0.05.

For decisions, the interval is more useful than the p-value. Its lower bound shows the worst plausible case for shipping. If the lower bound sits close to zero, the expected gain may be small even when the result is significant.

## Reading the result

### Is a significant result automatically a business win?

No, a significant result is not automatically a business win. Statistical significance says the difference is likely real, not that it is worth having.

For practical significance, look at three things. Check the lower bound of the confidence interval and the cost of building and maintaining the change. Then check value metrics such as revenue alongside the conversion rate.

A variant that brings more orders but lower revenue per order can be significant and unprofitable. The section on measuring in revenue in the [conversion rate optimization guide](https://alidemirbas.com.tr/en/blog/conversion-rate-optimization-guide) covers this in more detail.

If you see a surprisingly large lift, check the data first. Ronny Kohavi and co-authors call this Twyman's law in a 2014 paper: any figure that looks interesting or different is usually wrong. A surprising win should be treated as a possible measurement or setup error until shown otherwise. Whether [the traffic split matches the plan](https://alidemirbas.com.tr/en/blog/control-group-ab-testing), and whether both groups log the same events in the same way, are the first places to look.

### What does a non-significant result mean?

A non-significant result means the data is not strong enough to rule out zero difference. Scenario B shows this: the true difference could be −0.66 points or +1.86 points.

With a result like that, first check whether the test was planned with enough power. If it was, and the interval is narrow, you can say the effect is too small to matter. If the interval is wide, the result is inconclusive, and the idea needs a larger sample or a bolder change.

## Three mistakes that break a result

Even with the right calculation, the way a test is run can invalidate its result. These are the three common mistakes that do the most damage when interpreting A/B test results.

**Peeking and stopping early.** The p-value goes up and down as data arrives. Checking it repeatedly and stopping the first time it drops below the threshold pushes the false positive rate well above the level you set.

In [Evan Miller's well-known analysis of peeking](https://www.evanmiller.org/how-not-to-run-an-ab-test.html), if you look at a test ten times, what you think is 1% significance is actually just 5%. A 2017 KDD paper by Johari and co-authors also shows that continuous monitoring makes classic p-values and confidence intervals unreliable. It proposes "always valid" p-values that can be checked at any point.

Setting a stopping rule before the test, such as a planned sample size, avoids the problem. If you want to watch results continuously, check whether your tool uses sequential testing.

**Reading many metrics and variants at the same threshold.** Every comparison is another chance for a false positive. Say you read 10 independent metrics at the 0.05 threshold and nothing has changed. The chance that at least one comes out significant is about 40% (1 − 0.95¹⁰).

Pick one primary metric before the test and report the others as supporting. If you compare many variants, divide the threshold by the number of comparisons with a method such as the Bonferroni correction, which NIST also describes.

**Switching the metric or segment after the fact.** Suppose the primary metric is not significant. Declaring [a significant segment](https://alidemirbas.com.tr/en/blog/ab-test-segmentation) or a different metric to be "the real result" then recreates the multiple comparisons problem in a hidden form. A finding like that is a new hypothesis and needs its own test.

How to plan the hypothesis, metric and duration of a test from the start is covered step by step in the [complete A/B testing guide](https://alidemirbas.com.tr/en/blog/what-is-ab-testing-complete-guide). Significance is the last step of a well-planned test.