Replace statistical proof with converging evidence. When five different sources point to the same problem, you can fix it without a test.
The benchmark
Rank your evidence sources from fastest to slowest: requests received by customer support, return reasons, searches typed into your internal search engine, session recordings, a question asked after purchase, and five people observed buying on their phones.
The recording tool is almost always installed, and almost never watched. A founder told us this during a diagnostic:
"I've never managed to make use of what I saw on Clarity."
It took us a year to turn it into a routine: 45 minutes a week, five product pages chosen from those that convert below average or whose returns are rising, six sessions per page, three on desktop and three on mobile, with the sound off.
The rule
The less traffic you have, the more every change must answer an observed problem, and the bolder it must be.
A small tweak on small traffic can't be measured. A blocker spotted by three independent sources is more likely to be real than a 3% gain declared significant by a tool you check every morning.
The steps
- Customer support. List the ten most frequently asked questions: each one points to information missing from the site.
- Returns. Read the reasons for the twenty most-returned products.
- Sessions. Watch twenty cart abandonments, then twenty purchases. If a behavior shows up in 40% of abandonments, check that it doesn't also show up in 40% of purchases. If it does, it explains nothing.
- The five people. At a ready-to-wear brand, a test had won a 23% increase in add-to-carts, not orders yet, among new visitors. Before generalizing, five customers observed during a shopping session revealed a barrier the numbers had missed, summed up by the manager: "What they're missing is knowing how it looks on someone 5'3", how it looks in a size 12."
- Before and after. Change one thing only, date it, compare four full weeks before and four after, with no campaign in between, and keep a control metric that shouldn't move.
- The logbook. Record every change.
A sentence to say to your leadership, in your own words:
"We won't test. We'll fix what customer support, returns, and twenty sessions point to together, and we'll compare four weeks to four weeks."
The trap
The most common trap is believing twenty sessions are a sample. They're twenty anecdotes: they prove nothing, they make a hypothesis credible, and that's already a lot when you can't test.
Two other traps take one line each: copying the "best practices" of a brand that has neither your customers nor your prices, and comparing the week of the change to a promotion week to conclude what you hoped.
