Your order volume decides before your budget does. Without enough orders to run tests back to back, go with a short engagement rather than a retainer or a hire. With the volume, an agency starts fastest, a freelance engagement costs the least, and hiring makes sense at around twenty tests a year.
We run an agency. Keep in mind, as you read the table below, that we have a stake in your picking its first column.
The benchmark
Do the math in orders, the unit a test is measured in. A common floor comes from CXL, which says it generally ignores results with fewer than 250 to 350 conversions per variant.
A two-version test therefore needs 500 to 700 orders, and you need 700 to 1,000 a month to run one every three weeks.
That leaves the question of what a test this size can detect. Using the shortcut in our guide to test sample size, 250 to 350 conversions only detect an effect of at least 21 to 25%.
Yet in Convert's data on tests run with its tool in 2025, six in ten show a gain under 20%. To detect +10%, plan on about 1,600 orders per variant.
Here are the ballpark figures we gathered in 2026 (the start-up times are our estimates).
| Agency | Freelancer | In-house hire | |
|---|---|---|---|
| Annual cost | €36,000 to €60,000 excl. VAT | €33,000 to €45,000 excl. VAT for 5 days a month | €60,000 to €90,000 fully loaded |
| Start-up time | 2 to 4 weeks | 1 to 2 weeks, once you've found the person | Several months between the job posting and the first test |
| What you still cover | Little, if design and development are included | Often design, development and the testing tool license | Producing the variants, depending on your teams |
| Where the cost comes from | Rates listed online for a standard program, €3,000 to €5,000 excl. VAT per month (our survey). The package we recommended last spring for two to three tests a month sat at the high end of the range | €550 to €750 per day for an experienced profile, according to LiveMentor | €44,000 to €65,000 gross for an experienced or senior profile, according to SEYOS and Digital Movers, plus roughly 40% in employer contributions (our calculation) |
The rule
Our house rule sets three conditions for hiring: about twenty tests a year, enough design and development capacity to produce the variants, and someone who can make decisions with other teams without going through you.
Below that, a hybrid setup keeps coming up in our meetings: one person in-house who owns the topic, and an outside provider for the method and the production.
In June 2026, the founder of a brand who came to us for advice had arrived at this setup after trying everything else. To keep her e-commerce running, well beyond CRO alone, she had gone through “a string of freelancers,” then a “very broad” permanent position that didn't work out. Her conclusion: “maybe we're better off handing the e-commerce side to an agency,” with someone in-house for content and coordination.
I told her it was “probably a better deal than an employee,” then added: “Either way, we'd need to run the numbers.” The table above is a first version of that calculation, which I didn't have on hand.
This setup adds up two costs. In our view, it makes sense as long as the person who answers for the numbers stays with you, as the guide on the three-person team explains.
The steps
- Volume. Count the monthly orders that go through the page you want to improve, then divide the number of orders a test requires by that figure. That gives you the length of a test, in months.
- Production. Ask who produces the variants, design and development included, and who will actually work on your account.
- The losing test. Ask for a recent losing test and what was concluded from it, because a track record made up only of winners signals cherry-picking.
- Method. The length of a test must be calculated before launch, in conversions per variant and purchase cycles.
- Plan B. Ask what happens if your traffic doesn't allow testing. An honest answer describes different work, the kind covered in the guide to optimizing without testing.
- The exit. The contract should state who gets the testing tool account and the test history when the engagement ends.
The hypothetical example below applies the first step. Rewrite it with your own numbers for your situation:
“We get 400 orders a month. A test that can detect a 10% lift needs 3,200, or eight months. I'm proposing a short engagement for diagnosis and fixes, and revisiting a retainer or a hire once a test fits in a month.”
The trap
The costliest trap is signing a testing retainer without the volume. In September 2026, I told a brand's founders as much:
“Your traffic isn't enough for us to run A/B tests, the measurement cadence is too slow and you're going to feel like nothing is moving.”
At 150 orders a month, a store waits more than twenty weeks to detect a 20% effect (our calculation).
The other trap shows up in case studies, when a winning segment is showcased from an otherwise flat test. Slice enough segments after the fact and you'll almost always find one that wins.
Check that the segment was defined before launch, then gauge how plausible the number is. Convert advises rerunning, before anything goes live, the 7.8% of tests that report a gain of more than 100%.
