> ## Content Index
> Fetch the complete content index at: https://conversion.club/llms.txt
> Use this file to discover other available public pages before exploring further.

# How Many Sessions Do You Need to Run an A/B Test?
- URL: https://conversion.club/playbook/how-many-sessions-do-you-need-to-run-an-a-b-test/
- Published: 2026-10-01T20:19:28.000Z
- Updated: 2026-10-19T20:26:12.000Z
- Description: Far more than you'd think, and it all depends on the effect you hope to detect. For a 2% rate and a 10% improvement, plan on nearly 80,000 sessions per variant. For a 20% improvement, about 20,000.
- Author: Jeremie Werner
- Tags: Improve conversion, #playbook, A/B Testing, CRO

You need far more than you'd think, and it all depends on the effect you hope to detect. For a 2% rate and a 10% improvement, plan on nearly 80,000 sessions per variant. For a 20% improvement, about 20,000.

---

## The benchmark

The common rule of thumb, for 80% power and a 5% significance level: sessions per variant ≈ 16 × p × (1 − p) ÷ d², where p is your baseline rate and d is the target difference in absolute terms. Evan Miller's calculator gives the same orders of magnitude.

| Baseline rate | Target effect | Sessions per variant |
| ------------- | ------------- | -------------------- |
| 2%            | 2.0 to 2.2%   | 78,400               |
| 2%            | 2.0 to 2.4%   | 19,600               |

These are sessions on the tested page, not on the whole site. A handier shortcut for meetings counts conversions instead of sessions: about 16 ÷ ε² conversions per variant, where ε is the target relative improvement. For +20%, that's 422 orders per variant; for +10%, 1,614; for +5%, 6,304.

---

## The rule

Only launch a test if your traffic lets you reach a conclusion in [two to four full weeks](https://conversion.club/playbook/how-long-should-you-run-an-a-b-test-and-why-not-stop-early/), in seven-day blocks: 7, 14, 21 or 28, never 10 or 18\. Otherwise, test a bolder change, or [decide another way](https://conversion.club/playbook/how-do-i-optimize-my-site-without-enough-traffic-to-test/). 

A test sized to detect +20% will only catch a real +5% gain about one time in ten: the variant that truly delivers 5% will come out "not significant" nine times out of ten, and you'll bury it. 

Our consultant put it bluntly during a retrospective on our own tests: "A lot of our A/B tests come out 51 versus 49\. A lot of our tests are too small and too timid."

---

## The steps

1. Traffic. Count the weekly sessions on the page in question.
2. Minimum effect. Set the one that would justify the change, and accept that the test won't see anything below it.
3. Duration. Calculate it before you start, and write it down.
4. Discipline. Stick to that duration, whatever the tool shows. At a shoe brand, a test sent 2,000 visits per day to each variant and was still unreadable at thirteen days: the variant's daily conversion rate swung between 0.18% and 1.15%. It lacked duration, not traffic.
5. Failure rate. Announce it up front: two to three tests out of ten win, and that's normal.

A line to tell your leadership, using your own numbers: 

> "This test can only see a 20% gain. If it wins 5%, we won't know. We set the duration today, and we don't look before then."

---

## The trap

Stopping the test the day the tool declares a winner is the most common trap. If you look every morning, you'll eventually hit a lucky day, and you'll have validated chance. 

A simulation of 40,000 A/A tests, where both variants are identical, shows it: read once at the end, a test wrongly declares a winner 5 times in 100; read four times, 12.6 times; read every morning for four weeks, 27 times. 

The reverse trap exists too: testing a minor tweak with 30,000 visits a month. The test will run for a year.

#### Sources

- [Sample size calculator (Evan Miller)](https://www.evanmiller.org/ab-testing/sample-size.html?ref=conversion.club)
- [How not to run an A/B test (Evan Miller)](https://www.evanmiller.org/how-not-to-run-an-ab-test.html?ref=conversion.club)