Read it before you shelve it, because its confidence interval tells you the range the effect falls in.
Then look for the cause among four: faulty tracking, a test that's too small, an effect weaker than hoped, or a variant diluted in an audience that's too broad. And file it as “inconclusive” yourself, before someone else puts it on the wins board.
The benchmark
Look at the confidence interval before the verdict.
On one store, a test of the cart button label drew, in under two weeks, just over 14,000 visitors per version, with 106 orders for the control and 113 for the variant.
The tool showed +7.5%, with a 95% interval running from −17% to +40%.
A gain above +40% is very unlikely given this data, and the effect could just as well be negative: rolling it out carries a risk.
Between those two bounds, everything is still possible, so you can't conclude there's no difference either. The guide What does “95% significant” mean in an A/B test? explains what that 95% threshold guarantees.
The rule
A tight interval around zero settles the debate: the idea changes nothing useful, so move on to the next one.
A wide interval signals an underpowered test, and it's the case that comes up most often in our own analyses. Once the planned duration has passed, stop it and redo it differently, with a bolder change or on a more specific population.
The steps
- Tracking. Faced with a disappointing result, suspect the measurement first. In one of our first tests, I'd put the choice to the team this way: “either there's a measurement problem, or performance isn't where we expect it.” Start by comparing the size of the two groups with a chi-square test. By our calculation, a 49/51 split across 40,000 visits already falls outside chance, and Microsoft recommends not trusting an imbalanced test until the cause is found. Then hunt for a version shown somewhere it shouldn't be, an order counted twice, or a variant that loads after a visible delay.
- Power. A test sized to detect +20% catches a real +5% gain only about one time in ten, even though nothing is broken. The guide How many sessions do you need to launch an A/B test? gives the sizing formula.
- Effect size. At Google, LinkedIn and Microsoft, about two-thirds of promising ideas that are built and then tested don't improve the targeted metric, according to a paper co-authored by Ron Kohavi. In our view, a label or a color rarely moves the conversion rate enough for a test to see it.
- Segment. A test on all traffic averages out populations that react differently. In a hypothetical example, reassurance about delivery times gains 15% among new visitors, who account for a quarter of orders, and loses 3% among loyal customers. The average comes out at +1.5%, not significant. For the next test, state the target population in the plan before launch.
- Next steps. If your traffic doesn't allow a test with enough power, decide differently, using the body of evidence described in the guide How do you optimize your site without enough traffic to test?.
The trap
There are two ways to turn a non-significant test into a win, and the better known one is to slice the results after the fact until you find the segment that wins.
With ten independent slices, there's a four-in-ten chance that at least one clears the threshold by luck. Note that segment as a lead for the next test.
The other way is quieter, and we've done it. For another store, our analysis of a homepage test concluded after fourteen days: “no statistical conclusion possible.”
The new version was rolled out anyway, for a brand-consistency reason the test didn't measure, which is defensible. Yet our tracking database lists this test as “Win,” with no explanation. In six months, whoever consults it will read a measured gain.
The sentence to present
Before you close the test, write its conclusion in one sentence that contains the interval, the decision and the next step, then paste it into your tracking table in place of the status. Copy it, then rewrite it with your numbers for your leadership:
“This test doesn't let us conclude: the new label's effect falls somewhere between −17% and +40% in orders. I'm filing it as ‘inconclusive,’ neither won nor lost. I suggest retesting a bolder change, with a duration set before launch.”
