A/B test

An A/B test is an experiment in which site visitors are randomly split into two groups: one sees the current version of a page (A, the control), the other sees a modified one (B, the variant), and a metric chosen in advance shows which version performs better.

In plain words

Instead of arguing "which is better, this or that," you show both options to real shoppers at the same time and see which one gets more orders. The groups are formed at random and live under the same conditions (the same day, the same ads, the same prices), so the difference between them comes from the change itself, not from the season or a promotion.

Optimizely describes an A/B test as a way to compare two versions of a page or app: traffic is split randomly between the control and the variant, and statistical analysis shows which version reaches the goal better (Optimizely). NN/g adds that ideally the variant differs by a single design element (NN/g, A/B Testing 101). The method itself, as HBR reminds us, is almost a hundred years old, long predating websites (HBR, 2017).

The A/B test is the main tool for checking hypotheses in CRO.

Statistical significance in plain words

Even two identical pages will show different conversion over a week: 2.1% and 2.3%. That is chance. Statistical significance answers the question: how likely is it that the difference you see is not random?

Optimizely explains it through the p-value: the probability of seeing the same or a stronger result if there is actually no difference between the versions. The smaller the p-value, the less likely the lift is chance (Optimizely). The most common significance threshold is 95%; NN/g calls it the usual one (NN/g).

One caveat: "significant" does not mean "important for the business." NN/g reminds us that a significant result can be a false positive, a measurement error, or simply too small to matter in practice.

Sample size

How many visitors you need is decided before the test starts. NN/g names three inputs (NN/g):

1. Baseline conversion: the current conversion of the control version.
2. Minimum detectable effect (MDE): the smallest effect you want to notice, for example a 10% lift over the current level.
3. Significance threshold: usually 95%.

The lower the baseline conversion and the smaller the effect you want to detect, the more people you need. A VWO example: a page with 5% conversion, a 15% MDE and three versions (control plus two variants) requires 40,533 visitors per version under the classic (frequentist) approach and 14,674 under the Bayesian one (Wingify/VWO).

Evan Miller gives a rule of thumb for a quick estimate: n = 16σ²/δ², where σ² = p(1 − p) for conversion p, and δ is the minimum effect in absolute terms (Evan Miller).

Illustrative example. A store with 2% conversion (p = 0.02) wants to detect a rise to 2.4% (δ = 0.004). σ² = 0.02 × 0.98 = 0.0196. n = 16 × 0.0196 / 0.000016 ≈ 19,600 visitors per version, about 39,000 for the whole test.

Duration

Optimizely recommends running a test for at least one business cycle (7 days) and calculating the duration as sample size × number of versions / average visitors per day (Optimizely). NN/g recommends at least one to two weeks even with plenty of traffic, because people's behavior varies by day of the week.

In the example above, at 3,000 visitors a day: 39,000 / 3,000 ≈ 13 days, that is, two full weeks.

VWO warns against both tests that run too long and comparing periods with different outside conditions, such as sales and holidays (Wingify/VWO). For a Ukrainian store that could be Black Friday or disruptions in a delivery service.

Common mistakes

• Stopping early. A variant leads in the first days and the test gets switched off. VWO states plainly that this is not a reason to stop (Wingify/VWO); NN/g names stopping based on real-time monitoring as a common mistake.
• Peeking. Classical statistics requires fixing the sample size and not looking at the result until the end. In his 2010 article, Evan Miller showed by simulation that if you check the result after every observation and stop at the first "significance," a false win happens in 26.1% of cases instead of the stated 5% (this is the worst case but, according to the author, not a rare one; Evan Miller). This is mathematics, not data about shopper behavior, so it has not gone out of date. If you need to look at interim results, use sequential or Bayesian methods; some tools (for example, Stats Engine in Optimizely) are built specifically for continuous monitoring.
• Changing settings mid-test: goals, design, traffic allocation (Wingify/VWO).
• Too many segments. Optimizely warns that small segments give unreliable results and false-positive conclusions (Optimizely).
• Many changes at once. It is unclear what exactly worked, and a multivariate test needs even more traffic.
• Technical SEO mistakes. Optimizely advises using a 302 rather than a 301 redirect for test URLs and setting rel="canonical".

What has changed: AI and setup quality

In Optimizely's 2026 report on 173 thousand "real" experiments (2018–2026, more than 1,200 companies), the best predictor of a win is the quality of the test setup: correctly set up experiments won in 58% of cases, weak ones in 35%. A "real" experiment here means a running test with a control and at least one variant, each with at least 1,000 visitors (Optimizely, 2026). This is data from the tool vendor, not an independent study.

In the same report, customers who use Optimizely's AI features ran about twice as many experiments as those who do not; the authors say outright that this is an observation, not a proven causal link. The practical takeaway, without numbers: AI helps generate hypotheses and variants faster, but sample size, duration and stopping rules stay the same.

When you do not need an A/B test

• Low traffic. If the calculation says the test will take several months, it loses its point. NN/g names low-traffic pages as a case where an A/B test is not suitable.
• You need to understand "why." An A/B test shows what changed but does not explain the reasons; for that you need qualitative research (NN/g).
• An obvious bug. Payment does not work, or the form does not accept Ukrainian phone numbers: you fix that, you do not test it.

In such cases usability testing, session recordings, micro-conversions (there are more of them than orders) and period comparisons help.

FAQ

What is A/B testing? An experiment in which half of the visitors see the old version of a page and half see the new one, and the data shows which is better.

How long should an A/B test run? As long as it takes to collect the sample calculated in advance, and no less than one full week (preferably two).

How much traffic do you need to A/B test a site? It depends on conversion and the expected effect. Usually it is thousands or tens of thousands of visitors per version; a sample size calculator gives the exact number.

Can you stop the test if a winner is already visible? In the classical approach, no: it raises the chance of a false conclusion. Wait for the planned sample.

Related terms

• CRO (conversion rate optimization)
• Conversion rate
• Micro-conversion

Sources

• Optimizely. A/B testing (Optimization Glossary) — optimizely.com/optimization-glossary/ab-testing
• Optimizely. Statistical significance (Optimization Glossary) — optimizely.com/optimization-glossary/statistical-significance
• Optimizely. Lessons learned from 173K experiments, 2026 — optimizely.com/field-notes/guides/173000-experiments
• Tim Neusesser. A/B Testing 101. Nielsen Norman Group, 30.08.2024 — nngroup.com/articles/ab-testing
• Amy Gallo. A Refresher on A/B Testing. Harvard Business Review, 2017 — hbr.org/2017/06/a-refresher-on-ab-testing
• Evan Miller. How Not To Run an A/B Test, 2010 (simulation, mathematical fact) — evanmiller.org/how-not-to-run-an-ab-test.html
• Wingify (VWO). A/B Testing guide — wingify.com/ab-testing
• Wingify (VWO). Sample size (glossary), updated 24.02.2025 — wingify.com/glossary/sample-size