Often, they only measure the people who accepted cookies. When the testing tool, or the tool that reads its results, waits for consent, the sample shrinks and the test runs longer. The banner does your recruiting for you, and it holds back the share of traffic that buys the most.
The benchmark
The number to pull before every test is your consent rate, calculated across all visits.
At a brand the authors work with, about three weeks after the banner went live, the data consultant walked through the numbers in a meeting:
“About 40% just let it slide […] and out of the total, 28% of people accept cookies.”
Yet the meeting notes recorded “46% acceptance.”
Both figures describe the same measurement: 46% of visitors who respond to the banner, which is 28% of all visits since four in ten click on nothing. Only that second figure is useful for sizing a test.
This rate divides your usable traffic. Take the calculation from the sheet on test sample size: with a 2% conversion rate and a target gain of 20%, you need 19,600 sessions per variant, or 39,200 in total, and every one of them must be a consenting session.
At a 28% consent rate, you have to push 140,000 sessions through the test. On a page that gets 14,000 sessions a week (hypothetical volume), the test goes from a little under three weeks to ten.
The rule
Before launch, state which population the test measures.
Google Ads Help gives the order of magnitude: people who consent are generally two to five times more likely to convert than those who don't, with wide variation by industry.
A gain measured among them therefore can't be carried over as is to all of your traffic.
Never pick a winner between two variants based on a modeled conversion, the kind Google's Consent Mode supplies when consent is missing.
GA4's behavioral modeling estimates audience and session metrics, and it is missing from the BigQuery export, from sequence segments and from several explorations. Google Ads modeling links conversions to interactions with your ads.
As we read it, neither was designed to tell you which variant was shown to someone who declined cookies.
The steps
- The setup. Open your store in a private browsing window, decline the banner, then check in the browser inspector whether the testing tool still recorded an assignment (cookie or local storage). You'll know whether your tests cover everyone or only the consenting share.
- The rate. Pull it by device and by market. Depending on your settings, some traffic outside Europe sees no banner at all, so the tested population changes from one country to the next.
- The duration. Divide the sample size by this rate, then write the resulting duration into the test plan.
- The count. Read the test's conversions from orders rather than from purchases seen by GA4, which loses some orders when consent is missing. Intelligems, which the authors use, exports orders by test group.
- The legal basis. Before running the tool without consent or moving assignment to the server side, have your data protection officer or your legal counsel settle the question. The CNIL exempts certain audience measurement trackers from consent, under cumulative conditions, and we found no blanket exemption in its texts for an A/B test cookie.
The trap
A variant that changes consent is the sneakiest case.
We haven't yet observed it on any of the authors' tests and are inferring it from the setup: when the tool assigns everyone but the analysis only counts consenting visitors, a version that delays or hides the banner shifts the acceptance rate in one group only.
The two groups then no longer contain the same audience, and no alert warns you. The check is quick: if you set a 50/50 split and the analysis consistently counts more people on one side, find out why before you read the result.
The sentence to present
Take the first three readings on your next test, then present the consequence to your leadership starting from this sentence, rewritten with your own numbers:
“This test will only count visitors who accept cookies, which is 28% of traffic. It will run ten weeks instead of three, and its result will only hold for that population.”
