It means that if your two versions were identical, chance alone would produce a gap at least as large as yours fewer than 5 times in 100.
The number doesn't promise that the variant has a 95% chance of winning, or that 95% of your winners are real. As for the threshold itself, it's a convention, and the right level depends on what a mistake would cost you.
The takeaway
Behind that 95% is a p-value below 0.05, which boils down to one sentence. At 0.03, it reads like this: “if A and B converted the same, a gap at least this large would show up 3 times in 100.”
The American Statistical Association reminded everyone in 2016 in its statement on the p-value: it doesn't measure the probability that a hypothesis is true, and it says nothing about the size of an effect or the importance of a result.
Your tool doesn't always show a p-value.
Intelligems, which we use, relies on a Bayesian model and gives a “probability to beat control.” That figure estimates the chance that the variant does better, based on the vendor's model, and the translation above no longer applies to it.
We made this mix-up ourselves, in black and white: our internal methodology sheet defined the p-value correctly, then told readers to read it “in the ‘probability to Be Best’ field.”
The threshold sets the risk you accept of seeing a gap where there is none, and that choice has a price in traffic.
| Threshold | False alarms per 100 tests with no real effect | Traffic needed (100 baseline at 95%, 80% power) |
|---|---|---|
| 90% | 10 | 79 |
| 95% | 5 | 100 |
| 99% | 1 | 149 |
The rule
Choose the threshold before launch, based on the cost of a mistake. A reversible, low-cost change can be decided at 90%.
For a checkout redesign, aim for 99%. Inherited from the scientific research of the last century, 95% was never calculated for online retail. So you can question it, as long as you stop touching it once results are visible.
We haven't always followed this rule. Our methodology sheet tolerates a risk of 0.1, a 90% threshold, with at least 200 conversions.
Yet from January to September 2026, our meeting notes and documents state four decision thresholds, all read from the probability the tool displays: 80%, 85%, 90% and 95%.
The threshold drifted depending on the person and the moment, because it was never tied once and for all to the cost of a mistake.
The steps
- The display. Check your tool's documentation to see whether it gives a p-value or a probability to beat control, and note it next to the threshold you chose.
- The interval. Read the range of effects compatible with your data before the verdict. In a made-up example, a gain between +1% and +30% and another between +14% and +16% both come out “significant,” yet they don't call for the same decision.
- The sample. Check that the planned size has been reached, because a p-value read mid-test no longer has the stated property (the guide on test duration explains why).
- The lower bound. Compare the bottom of the interval to the cost of rollout, development included.
To copy for your leadership, then rewrite with your own numbers, when the most cautious gain doesn't pay back the development:
“The test is significant, but in the worst case compatible with the data, it earns less than it costs to build. We're not rolling it out.”
The trap
The trap is believing that a testing program run at 95% is wrong only 5 times in 100. Those 5% apply to tests with no real effect, while the useful question is about your declared winners: how many are noise?
It all depends on the share of your ideas that work. According to the success rates compiled by Kohavi, Deng and Vermeer, one idea in three improves the target metric at Microsoft, and one in ten at Booking, Google Ads or Netflix.
Run the numbers on 100 tests, 10 of which carry a real effect. Properly sized, at 80% power, your tests detect 8 of them. The other 90 produce 4.5 false alarms, half of them in favor of the variant.
You announce about ten wins, and 22% of them are noise, even with a flawless setup. Moving to 99% brings that share down to roughly 5%, and each test then needs half again as much traffic.
Staying at 95%, one winning idea in three is enough to bring it down to 6%. In our view, the surest driver is still testing better ideas on large enough samples, and calculating the sessions you need happens before launch.
