How to calculate sample size for an A/B test

Sample size is the number of visitors each variant of an A/B test needs so you can reliably detect the effect you care about. You calculate it before the test starts, and it decides how long the test will run, or whether it is worth running at all.

The calculation rests on four inputs: the baseline conversion rate, the smallest effect you want to detect, the confidence level and statistical power. This guide explains the formula and works through a numeric example step by step. It also shows the mistake that changes the answer the most: mixing up relative and absolute effects.

To run the numbers on your own data, use the A/B test sample size calculator. The examples here use the same formula and constants as that calculator.

To calculate sample size for an A/B test, you use the classic formula for comparing two proportions. It finds the smallest group that keeps two risks in check at once.

One is a false alarm when there is no real difference (alpha). The other is missing a real effect (beta). The version the calculator on this site uses is:

Sample per variant = 2 × (z_alpha + z_power)² × p × (1 − p) ÷ (p × MDE)²

Here p is the baseline conversion rate. MDE (minimum detectable effect) is the smallest relative effect you want to detect. The z_alpha term is the z-value for your confidence level (1.96 for 95% with a two-sided test). Its counterpart z_power is the z-value for your power (0.8416 for 80%).

The p × MDE term in the denominator converts the relative effect into an absolute difference. This version approximates the variance of both groups with the control rate. A slightly more precise version uses each group's variance separately and returns a somewhat larger number, about 7% larger in the worked example below.

For a quick estimate, there is a rule of thumb from a 2009 paper by Ronny Kohavi and co-authors. At 95% confidence and 80% power, the sample per variant is about 16 × σ² ÷ Δ². For a conversion rate the variance σ² is p × (1 − p), and Δ is the absolute difference you want to detect.

The paper suggests replacing 16 with 21 to raise power to 90%. The details are in Kohavi and co-authors' guide to controlled experiments.

What inputs do you need?

The formula has four inputs, and each pushes the result in a different direction. The table summarizes them, the values commonly used in A/B testing and their effect on the sample.

The strongest lever in the table is the MDE, because it enters the denominator squared. The baseline rate matters too.

With a fixed relative MDE the formula works roughly in inverse proportion to the rate. To detect the same 15% lift at a 2% rate instead of 4%, the sample per variant rises from 16,745 to 34,187, about double. Expect tests on low-converting funnel steps to run much longer for that reason.

Take the baseline rate from the recent history of the same page and the same traffic rather than guessing. Most teams keep confidence and power fixed and choose the MDE per change.

When picking the baseline rate, use the audience the test will actually include. If the test runs only for mobile visitors, or only for people who reach the checkout step, take the rate from that audience's history. A sitewide average can differ a lot from the tested page's rate and make the sample too small or too large. Leaving out unusual periods such as sales, promotions and holidays also gives a more realistic starting value.

A worked example

Say a checkout page converts at 4% and you want to detect a 15% relative lift, with 95% confidence (two-sided) and 80% power. The calculation goes like this:

1. Find the absolute difference: 0.04 × 0.15 = 0.006. The goal is to detect a move from 4% to 4.6%.
2. Add the z-values and square them: (1.96 + 0.8416)² = 7.849.
3. Compute the variance term: 0.04 × (1 − 0.04) = 0.0384.
4. Apply the formula: 2 × 7.849 × 0.0384 ÷ 0.006² = 2 × 7.849 × 0.0384 ÷ 0.000036 ≈ 16,744.5.
5. Round up: 16,745 visitors per variant, or 33,490 in total for a two-variant test.

Kohavi's rule of thumb gives 16 × 0.0384 ÷ 0.000036 ≈ 17,067 for the same example, close enough for a first estimate. Divide the total by the eligible daily traffic to the page and you get the test duration in days. Rounding up to full weeks makes sure the test covers both weekday and weekend behavior.

Depending on the tool, MDE is defined as relative or absolute, and the difference can change the answer by orders of magnitude. A relative MDE is a percentage change from the baseline: at a 4% rate, a 15% relative MDE means 4.6%. An absolute MDE is a difference in percentage points: at the same rate, a 15-point absolute MDE means 19%. The calculator on this site uses a relative MDE.

Mixing them up has a large effect. Suppose you enter a 15-point absolute MDE instead of a 15% relative one in the example above. The formula then returns only 27 visitors per variant, against the 16,745 actually needed. A test run on a sample that small has almost no chance of detecting a realistic lift.

Tools for A/B testing disagree on the definition. Optimizely defines MDE as the minimum relative change in conversion rate. In its example, a 10% MDE at a 20% baseline covers 18% to 22%.

GrowthBook also uses a relative percentage improvement. Evan Miller's widely used calculator has a toggle between relative and absolute, and its default is absolute. Before comparing results across tools, check which definition each one uses. Also note that Optimizely's own calculator is built for its sequential stats engine, so its numbers are not directly comparable with fixed-sample formulas.

When choosing the MDE, start from the business side rather than the statistics: what is the smallest improvement that would justify shipping the change? That number comes from the gain needed to cover the cost of building and maintaining it.

Statistical power is the probability that a test detects an effect that really exists. With 80% power, if the true lift equals your MDE, the test will find it significant four times out of five. The remaining 20% is the Type II error, the risk of a false negative. Type I and Type II errors are explained in more detail in the guide to statistical significance in A/B testing.

The 80% figure goes back to the statistician Jacob Cohen. In a 1992 Psychological Bulletin article, he described .80 power as a convention proposed for general use. It is a common default, not a requirement. When missing a real effect would be expensive, 90% power makes sense.

In the example above, raising power to 90% increases the sample per variant from 16,745 to 22,418.

Seeing how sensitive the formula is to each input is the most useful part of planning. The table changes one input of the worked example at a time and keeps the rest fixed.

The MDE rows matter most: halving the effect quadruples the sample, and doubling it cuts the sample to a quarter. That is why, on a low-traffic site, testing a bolder change is usually more productive than testing a small change for a long time.

How does sample size change with A/B/n variants?

With more than one variant, each one needs its own sample, so total traffic grows in proportion to the number of groups. Optimizely's documentation shows this as a simple multiplication: 8,000 visitors per variation and 4 variations make 32,000 visitors.

A second effect comes on top. Comparing several variants against the control group raises the false positive risk. To keep it in check, alpha is often divided by the number of comparisons (the Bonferroni correction).

In an A/B/C test that compares two variants with control, alpha drops to 0.025 and the sample per variant rises from 16,745 to about 20,300. With three groups, the total need grows from 33,490 to about 60,800.

How does a one- or two-tailed test affect sample size?

A one-tailed test needs a smaller sample for the same alpha and power, because the formula uses 1.645 instead of 1.96. In the example that brings 16,745 down to 13,191. But a one-tailed test does not check for the variant making the metric worse. Choosing a direction to shrink the sample changes what the test measures.

The first risk of an underpowered test is missing a real effect. As power drops, "not significant" comes up more often and ideas that work get thrown away.

The second risk is less well known. A 2014 paper by Andrew Gelman and John Carlin shows a problem with low power. When power falls much below 0.5, statistically significant estimates tend to be much larger in magnitude than the true effect. The authors call this a Type M (magnitude) error and work through examples in the paper.

In practice, a test that finds a "winner" on a small sample can overstate the lift. The shipped change then underdelivers and the team tries to work out why. Sample size planning protects not only whether a result is real but how big it really is. Stopping a test before it reaches the planned sample creates the same problem.

A survey sample size tells you how many people you need to estimate a single proportion within a margin of error. An A/B test sample size tells you how many you need to detect a difference between two proportions. The survey formula is n = z² × p × (1 − p) ÷ e², where e is the margin of error. At 95% confidence and a ±5-point margin, with the most cautious assumption p = 0.5, the answer is 385 people.

That 385 says nothing about an A/B test. An A/B test looks for a small difference between two groups, not the rate itself.

The smaller the difference, the larger the sample. It reaches tens of thousands per group, as in the example above. Planning an A/B test with a number from a survey calculator starts the test underpowered from day one.

Sample size is one part of planning a test. How to plan the hypothesis, metric and the rest of the test is covered in the complete A/B testing guide.