A/B test segmentation: when is a segment win enough to roll out?

A/B test segmentation shows you what an average result hides. The same tool can also make a variant with no real effect look like a mobile winner. Segments matter for both reasons: set up well, they tell you who a change works for. Set up badly, they dress chance up as a finding.

This post covers the decision side of segments. It shows which segments to write down before a test and why slicing results by many segments multiplies false wins. It also covers how smaller samples per segment cut power, how Simpson's paradox appears, and when a segment-specific rollout is defensible. One hypothetical example runs through all of it.

Segmentation in an A/B test does one of two jobs. It either targets the test at a chosen audience from the start, or it splits a finished test's results into subgroups. The first is pre-segmentation and the second is post-segmentation.

Randomization and hypotheses are covered in the complete guide to A/B testing. This post is about how a segment enters a decision.

An A/B test measures one average effect across everyone in it. That average can make a change that works in opposite directions in two segments look like no change at all. Finding that hidden difference is the value of segmentation. The risk sits in the same place: the more slices you look at, the likelier one of them differs by chance.

Pre-segmentation and post-segmentation

In pre-segmentation, the segment is defined before the test. Either the test is shown only to that segment, or the segment goes into the analysis plan and gets reported whatever it shows. In post-segmentation, device, source or country filters are tried after the results arrive.

The two do not carry the same weight as evidence. The European Medicines Agency (EMA) 2019 guideline on subgroup analysis ties a subgroup finding's credibility to three things. Those are a well-founded definition made in advance, a plausible mechanism and replication.

It adds that merely listing a factor for later analysis does not count as pre-specification.

The same guideline says a missing pre-specification is not by itself a reason to dismiss a finding. A case against a finding should rest on plausibility and replication. The guideline was written for clinical trials and is not a standard for product experiments. Its questions about what a finding rests on still help when you read a product test.

Optimizely's own segmentation documentation takes the same line, as of November 3, 2026. It says segments and filters should be used for data exploration, not for making decisions. A segment found after the fact is therefore a hypothesis for the next test, not a result.

A segment must be known before assignment

A segment is only valid if the variant cannot change who belongs to it. Device type, traffic source on the first visit and new versus returning status at test start are all fixed before assignment. Groups such as "added to cart", "clicked the new widget" or "engaged users" can be changed by the variant itself. The EMA guideline also says variables measured after treatment are not usually appropriate for defining subgroups.

Suppose variant B sends more people to the cart. B's "added to cart" group then holds a different mix of people from A's. A comparison inside that group no longer has randomization behind it.

Freezing each user's segment when they enter the test removes the problem. You may already score customers by RFM (recency, frequency, monetary). If so, use each customer's group on the day the test starts. How the score is built is covered in the post on RFM analysis.

Splitting results by segment means running a separate significance test for each segment. Each test carries its own chance of a false positive, and their build-up is the multiple comparisons problem. How this grows across ten metrics is covered in the post on statistical significance in A/B testing.

Segments add two problems. The number of slices grows without anyone noticing, and teams usually count only the winning direction as a finding.

Hypothetical example: how many winners appear when no segment has an effect?

Assume the variant has no effect on anyone. The team reads results at a 5% significance level with a two-sided test. In a two-sided test, half the false positives favour the variant and half go against it.

The table shows the purely chance findings by the number of segments examined. It assumes segments are independent. Slices of one dimension (mobile and desktop) are independent, but overlapping dimensions (mobile and new visitors) are not, so the figures are approximate.

Twelve segments add up fast from four dimensions: device (2), visitor type (2), traffic source (4) and region (4). If those twelve tests were independent, about one test in four would show at least one segment "winner" even when nothing works. Device, source and region slices share the same users, so the real rate can differ. Read 26.2% as an illustration.

The figure that needs no independence is the expected number of false winners. Each segment test has a 2.5% chance of a false win, so the count is 12 × 0.025 = 0.3 per test. A team that runs ten such tests reports about three false segment wins on average, with no real effect anywhere.

Bonferroni correction or a follow-up test?

The sources part ways here. The NIST statistics handbook describes the Bonferroni method: run each of g comparisons at α/g to keep the overall false positive risk under α. Here g is the number of comparisons actually tested, and the last column of the table shows that threshold. NIST frames the method for a group of comparisons selected in advance.

The EMA, by contrast, does not recommend adjusting the threshold for exploratory subgroup analyses. Its reason is that such analyses only signal whether further investigation is needed. Instead of correction, it asks whether the mechanism is plausible and whether the finding replicates in other data.

The two positions answer different questions more than they conflict. If you will decide on a segment, use a corrected threshold over a fixed number of planned segments. If you are exploring, skip the correction and confirm the finding in a new test.

An A/B test's sample is usually sized to detect an effect across all traffic. Split into segments, each slice has a smaller sample. Its chance of detecting a real effect, its statistical power, drops.

How sample size is worked out is covered in the post on how to calculate A/B test sample size. To try your own numbers, use the sample size calculator.

Hypothetical example: power in each segment

The main example in this post has 50,000 users per arm, and 60% of traffic is mobile. Mobile users convert at half the desktop rate. The figure in brackets is the control conversion rate. The table shows the chance of detecting a 10% relative lift at a 5% significance level.

A test that looks reasonable on all traffic misses a real 10% mobile lift about three times in five.

A significant result in a low-power segment has its own problem. This follows from how the threshold works rather than from a cited study. Only estimates that land above the true effect can clear the bar, so a significant estimate tends to overstate the effect. If you want to decide on a segment, size that segment's sample before the test and extend the test if needed.

The table uses a 5% significance level. If the decision will use a corrected threshold, size the sample with that threshold too. If the mobile decision uses 0.025, as below, 80% power needs 97,706 users per arm instead of 80,682.

The interaction test has even less power

The real question is usually not "is there an effect on mobile?". It is "is the mobile effect different from the desktop effect?". An interaction test answers that second question, and it has less power.

Brookes and colleagues ran a simulation study in 2004. A trial with 80% power for the overall effect had only 29% power for an interaction of the same size. Keeping the same power took a fourfold larger sample.

That fourfold figure holds for an interaction as large as the overall effect. The study adds that the sample needed rises sharply for smaller interactions.

For a hypothetical test with two equal segments, a direct calculation gives the same result: 28.8% power and four times the sample. The reason is arithmetic. Each half has a standard error about 1.4 times the overall one, and the difference between the halves has twice the overall one.

The same calculation gives 16 times the sample for an interaction half the size of the overall effect. So do not read four times as an upper bound. Size the interaction sample separately, from the interaction size you expect and the traffic split between segments.

Simpson's paradox is when a difference that points the same way in every segment reverses once the segments are combined. In a randomized test with a fixed split, both arms have the same expected segment mix, so no reversal is expected. The paradox usually appears when the two arms' mix drifts apart. The commonest way is changing the traffic split partway through the test.

Kohavi and Longbotham's paper on unexpected results, drawn from Microsoft experiments, gives a real case. In one experiment the Treatment looked 4% worse overall but beat the Control on almost every day.

The Treatment had 1% of users for 26 days, then 5% for a week, then 50% for the last two days. The test started on a Monday, so those last two days fell on a weekend, when clicks per user run lower. With so large a Treatment share on those days, they pulled its average down out of proportion.

Hypothetical example: the split changes mid-test

The variant gets 10% of traffic in week one and 50% in week two. A campaign lifts conversion in week one. The variant beats the control by 5% in both weeks.

The total row shows the variant a quarter worse, because most control users come from the high-converting week. Weight both weeks equally and the control is 3.00%, the variant 3.15%, and the difference is +5% again. The fix matches Kohavi's advice. Keep the Treatment to Control ratio fixed during the test and finish any ramp-up before it starts.

How to tell Simpson's paradox apart in your own data

Your own data needs three separate checks, and none replaces another.

The first is the sample ratio mismatch (SRM) check. It asks whether the user counts in each arm differ from the planned split, say 50/50, by more than chance. A statistical test such as chi-square answers that. Microsoft Research's article on diagnosing sample ratio mismatch treats it as a check every test must pass before its effects are analysed.

The second check repeats the same test inside each segment and each period in which the split changed. The Microsoft article also suggests checking whether an SRM shows up only in some segments or everywhere. The basics are covered in the post on the control group in A/B testing.

The third check puts the two arms' segment mix side by side. Random assignment does not make the shares exactly equal, and small gaps are normal. What you look for is a systematic gap that chance cannot explain.

Hypothetically, take 50,000 users per arm with mobile at 60% of control but 52% of the variant. That gap is far beyond chance, and the total cannot be trusted until you find the cause.

These checks do not cover each other. A planned 50/50 split can run 60/40 while both arms still have the same mobile share. The reverse also happens: the overall ratio holds while the segment mix drifts. The EMA guideline also notes that the balance randomization gives may not fully hold inside subgroups.

If all three checks are clean and the total still conflicts with a segment, you are most likely not looking at Simpson's paradox. You are looking at a difference between segments, real or chance. The interaction test in the next section tells those apart.

These checks show that a shift happened but not why. For the cause, look at assignment logs and the dates of any split changes.

This hypothetical example follows a product page change tested with 50,000 users per arm. The primary metric is the share of users who placed at least one order during the test, a user-level conversion rate. The p-values in the table are computed for that yes/no metric. Total orders divided by users cannot be tested this way, because users with several orders change its uncertainty.

For a clean example, the split stays at 50/50, the SRM check is clean and both arms have 30,000 mobile users.

Overall, the variant is 0.01 points behind and the difference is not significant, so the test shows no overall difference. On mobile it looks like a significant winner with a 12.5% relative lift.

The 95% confidence interval for the mobile difference runs from +0.02 to +0.48 points. It excludes zero, but its lower bound sits close to it. The first reading is "keep the old page on desktop and ship the new one on mobile". Real tests whose first reading turned out wrong are collected in the post on A/B testing examples.

With a fixed split, the total difference equals the weighted average of the segment differences (0.6 × 0.25 + 0.4 × (−0.40) = −0.01 points). So if the total estimate is behind while the mobile estimate is ahead, the desktop estimate must be further behind.

The interaction test: do mobile and desktop really differ?

A significant result in one segment and a non-significant or opposite one in another does not prove the segments differ. Gelman and Stern's 2006 paper says it in its title. The difference between "significant" and "not significant" is not itself statistically significant.

In their example, one estimate is significant (25 ± 10) and the other is not (10 ± 10). Yet the difference between them is 15 ± 14, nowhere near significant.

The right test looks at the difference between the two segment effects. Altman and Bland's note on interaction in the British Medical Journal (BMJ) gives that difference a standard error. It equals the square root of the sum of the squared standard errors.

In the example, the mobile and desktop effects differ by 0.65 points, with a standard error of 0.225 points. That gives z = 2.89 and p = 0.0038. You can get each segment's p-value with the A/B test significance calculator. The interaction still needs its own test on that difference.

How robust is the decision?

Two assumptions drive the decision: how many mobile variant users bought and how many segments were examined. In the table, the desktop figures, arm sizes and control rates stay fixed. Only the number of mobile variant buyers changes.

If device was written into the plan, the Bonferroni threshold is 0.025 for two segments and 0.05 for a single interaction hypothesis. If twelve segments across four dimensions were scanned after the results came in, the segment threshold becomes 0.05 / 12 = 0.0042.

The interaction threshold depends on how many interaction hypotheses were tested. One overall test per dimension makes four hypotheses and a 0.0125 threshold. Comparing every pair of segments within each dimension makes 14 comparisons and 0.0036. An overall interaction test for a four-category dimension is not the same as its pairwise comparisons.

In the main scenario (675), the interaction clears the planned 0.05 and the four-test 0.0125. It narrowly misses the 0.0036 of 14 pairwise comparisons.

The mobile effect itself does not clear 0.025. The 97.5% confidence interval that matches that threshold runs from −0.01 to +0.51 points, so it includes zero.

The decision this data supports is to keep the control on desktop and run a separate mobile test with the hypothesis written in advance. Had the mobile variant reached 700 orders, a planned analysis would put the mobile effect under its threshold. Had the same figure come from twelve segments scanned after the fact, it would miss 0.0042. The finding would then be a strong lead, and the cautious path is to replicate it in a test targeted at mobile.

Turning a segment win into a rollout opens the variant to that segment only, and it means maintaining two experiences for good. The list below is not a universal rule. It is this post's cautious decision framework for moving from a test on all traffic to a rollout in one segment:

• The segment was written down before the test and defined by a trait known before assignment.
• The interaction test clears a threshold corrected for the number of interaction hypotheses tested.
• The segment's own effect clears its own corrected threshold, and the matching confidence interval has a lower bound that matters to the business.
• A plausible mechanism explains the difference, such as a shorter form fitting a mobile screen.
• The SRM check is clean overall and inside the segment, and the arms' segment mix shows no gap that chance cannot explain.
• The cost of running two versions is smaller than the expected gain.

A segment-specific rollout also needs a tool that can target that segment. For Shopify stores, the post on A/B testing on Shopify covers which route can split visitors by device or segment.

If the test targeted one audience from the start, the interaction condition does not apply. The question there is whether the variant works for that audience, and a significant difference from another segment is not required.

When moving from all traffic to one segment, if any condition is missing, the next step is usually the same. Write the finding down as a hypothesis and run a new test targeted at that segment only. That second test gets its own sample size calculation.

As testing scales, this becomes part of a conversion rate optimization process. Segments found after the fact feed the next hypothesis list.

A segment plan is a short document, written before the test, that fixes which slices you will read, by which rules and in which unit. In the example, the unit is the user, segmented at first entry. The metric is whether that user ordered at least once in the test. You can build the plan in this order:

1. Write a mechanism for each segment. "Completion rises on mobile because the form gets shorter" is a hypothesis. "Let's check mobile too" is not.
2. Pick segment dimensions from traits known before assignment, and freeze each user's segment at first entry.
3. Limit the number of segments and write down the decision threshold now, for example with Bonferroni.
4. Size the sample separately for every segment you will decide on, at the corrected threshold the decision will use. Size the interaction separately too: one half the size of the overall effect needs about 16 times the users.
5. Write the decision rule: interaction test, segment effect, smallest meaningful lift and the cost of two versions.
6. Keep the traffic split fixed for the whole test, and run the SRM check overall and in every segment.
7. Label every slice outside the plan as exploratory and report it whatever it shows.

A plan like this turns segment analysis from a search for a story after the results into a decision made before the test. When the interaction and the segment effect both clear the threshold in a planned segment, a segment-specific rollout is a defensible call. Findings that come close but miss the threshold are often the start of a well-defined next test.