You need far more than you'd think, and it all depends on the effect you hope to detect. For a 2% rate and a 10% improvement, plan on nearly 80,000 sessions per variant. For a 20% improvement, about 20,000.
The benchmark
The common rule of thumb, for 80% power and a 5% significance level: sessions per variant ≈ 16 × p × (1 − p) ÷ d², where p is your baseline rate and d is the target difference in absolute terms. Evan Miller's calculator gives the same orders of magnitude.
| Baseline rate | Target effect | Sessions per variant |
|---|---|---|
| 2% | 2.0 to 2.2% | 78,400 |
| 2% | 2.0 to 2.4% | 19,600 |
These are sessions on the tested page, not on the whole site. A handier shortcut for meetings counts conversions instead of sessions: about 16 ÷ ε² conversions per variant, where ε is the target relative improvement. For +20%, that's 422 orders per variant; for +10%, 1,614; for +5%, 6,304.
The rule
Only launch a test if your traffic lets you reach a conclusion in two to four full weeks, in seven-day blocks: 7, 14, 21 or 28, never 10 or 18. Otherwise, test a bolder change, or decide another way.
A test sized to detect +20% will only catch a real +5% gain about one time in ten: the variant that truly delivers 5% will come out "not significant" nine times out of ten, and you'll bury it.
Our consultant put it bluntly during a retrospective on our own tests: "A lot of our A/B tests come out 51 versus 49. A lot of our tests are too small and too timid."
The steps
- Traffic. Count the weekly sessions on the page in question.
- Minimum effect. Set the one that would justify the change, and accept that the test won't see anything below it.
- Duration. Calculate it before you start, and write it down.
- Discipline. Stick to that duration, whatever the tool shows. At a shoe brand, a test sent 2,000 visits per day to each variant and was still unreadable at thirteen days: the variant's daily conversion rate swung between 0.18% and 1.15%. It lacked duration, not traffic.
- Failure rate. Announce it up front: two to three tests out of ten win, and that's normal.
A line to tell your leadership, using your own numbers:
"This test can only see a 20% gain. If it wins 5%, we won't know. We set the duration today, and we don't look before then."
The trap
Stopping the test the day the tool declares a winner is the most common trap. If you look every morning, you'll eventually hit a lucky day, and you'll have validated chance.
A simulation of 40,000 A/A tests, where both variants are identical, shows it: read once at the end, a test wrongly declares a winner 5 times in 100; read four times, 12.6 times; read every morning for four weeks, 27 times.
The reverse trap exists too: testing a minor tweak with 30,000 visits a month. The test will run for a year.
