> ## Content Index > Fetch the complete content index at: https://conversion.club/llms.txt > Use this file to discover other available public pages before exploring further. # How to A/B Test on Shopify, With or Without an App - URL: https://conversion.club/playbook/how-to-a-b-test-on-shopify-with-or-without-an-app/ - Published: 2026-10-02T07:49:18.000Z - Updated: 2026-10-19T20:25:41.000Z - Description: Without an app, you create an experiment in Shopify Rollouts, included from the Grow plan: half your traffic sees your modified theme, the other half sees the current one. With an app, you mostly pay for what Shopify doesn't provide: the calculation that tells you whether the gap is real. - Author: Jeremie Werner - Tags: Improve conversion, #playbook, A/B Testing, Shopify Without an app, you create an experiment in Shopify Rollouts, included from the Grow plan: half your traffic sees your modified theme, the other half sees the current one. [With an app, you mostly pay for what Shopify doesn't provide](https://conversion.club/playbook/how-much-do-your-shopify-apps-cost-each-year/): the calculation that tells you whether the gap is real. --- ## The benchmark A tutorial that still recommends Google Optimize is more than three years old, since the tool [has been unavailable since September 30, 2023](https://support.google.com/analytics/answer/12979939?hl=fr&ref=conversion.club). The current options are listed below, at public prices as of early October 2026, billed monthly. | Option | Price | What it tests | Main limitation | | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------ | | Shopify Rollouts | included, [Grow plan](https://www.shopify.com/fr/tarifs?ref=conversion.club) required (€105/month) | theme, checkout and accounts, catalogs (prices and availability), discounts | displayed rates only, no documented significance calculation | | [Shoplift](https://apps.shopify.com/shoplift?ref=conversion.club) | $99 to $999/month | themes, templates, URLs, prices (Advanced plan) | billed on all site traffic, tested or not | | [Intelligems](https://www.intelligems.io/pricing?ref=conversion.club) | from $69/month, depending on orders and the modules you choose | content and themes, then prices, discounts and shipping through modules | pricing module has no free trial, three-month commitment | | [Visually](https://apps.shopify.com/visually-io?ref=conversion.club) | $15 to $80/month depending on orders | pages, cart, checkout, upsells | 600 orders a month at most for $80 | | [Convert](https://www.convert.com/pricing/?ref=conversion.club) | $399/month | the whole site | high entry price | | [GrowthBook](https://www.growthbook.io/pricing?ref=conversion.club) | free for up to 3 people, then $40 per seat | the whole site | integration needs a developer | --- ## The rule Choose based on what you want to test, then count the hours before comparing subscriptions. According to [Shopify's help center](https://help.shopify.com/fr/manual/markets/rollouts/rollout-types?ref=conversion.club), you can test a theme, a checkout page, a catalog and its prices, or a discount without an app, while shipping rates require one. We found those hours out along the way. Our theme tests run through an app and a full copy of the theme. In summer 2025, on one of the first, our developer summed up the constraint: > "If you make a change for a bug fix, you have to push it to both things at the same time. That's where it gets a bit annoying to manage." We estimate that preparing a test takes between two and eight hours depending on the component. As for the losing copy, it stays in place after the last day, since Intelligems, which we use, [recommends keeping it for at least two months](https://docs.intelligems.io/content-experiences/content-experiences-overview/how-to-set-up-a-theme-change-experience/ending-a-theme-test?ref=conversion.club): without it, anyone already assigned to that version would hit an error. In our view, the value of Shopify's tool lies there more than in the price. With Rollouts' "Edit theme" option, the unmodified areas stay synced with the published theme, [according to Shopify](https://help.shopify.com/fr/manual/markets/rollouts/create-a-rollout?ref=conversion.club), so a fix outside the tested area would only need to be applied once. --- ## The steps We haven't run any test with Rollouts yet, so these steps follow its documentation ([creating](https://help.shopify.com/fr/manual/markets/rollouts/create-a-rollout?ref=conversion.club) and [managing](https://help.shopify.com/fr/manual/markets/rollouts/managing-rollouts?ref=conversion.club) a rollout), including the French labels. 1. The rollout. In the admin, open Markets > Rollouts (or Markets > Déploiements, depending on the page), click "Create a rollout" and choose the "Experiment" type. The "Launch" type applies changes permanently, with no control group. 2. The change. Limit yourself to one. "Edit theme" suits anything you can adjust in the editor, while "Replace current theme" pits a second, complete theme against the first, best reserved for bigger projects like navigation. 3. The dates. By default, Shopify suggests a 50/50 split and an end date 90 days after the start. Replace that deadline with the one your [sample size](https://conversion.club/playbook/how-many-sessions-do-you-need-to-run-an-a-b-test/) gives you, as [the guide on test duration](https://conversion.club/playbook/how-long-should-you-run-an-a-b-test-and-why-not-stop-early/) explains. 4. The stop. That same summer of 2025, the day a problem came up, nobody on our team knew where the pause button was in our tool. Before you launch, find "Pause" and "End." 5. The calculation. On the end date, pull the sessions and orders for each group, then calculate the confidence interval. 6. The decision. "End" makes you choose between "Discard changes" and "Apply changes," and once a change is applied, it can't be undone. --- ## The trap The trap is reading the numbers Rollouts displays and [declaring a winner](https://conversion.club/beyond-a-b-testing-tailoring-your-shopify-site-to-your-audiences/). For a theme experiment, [the help page on analytics](https://help.shopify.com/fr/manual/markets/rollouts/analytics?ref=conversion.club) lists four rates: conversion, bounce, reached checkout and add to cart. It mentions no significance calculation, no revenue per visit, no export and no group sizes. So check on day one that you can pull sessions and orders for both sides, because no interval can be calculated without those numbers. To read that interval, see [the guide on what "significant at 95%" means](https://conversion.club/playbook/what-does-95-significant-mean-in-an-a-b-test/). That leaves consent, which is absent from the five Rollouts help pages we read. In its [July 2025 self-assessment tool](https://www.cnil.fr/sites/default/files/2025-07/outil%5Fd%5Fauto-evaluation%5Fmesure%5Fd%5Faudience.pdf?ref=conversion.club), the CNIL excludes from the audience-measurement exemption "any creation of user cohorts to show them differentiated content," whether or not the cohort is drawn at random. Our reading, which your DPO should confirm, is that an A/B test falls under this. If yours waits for consent, your acceptance rate factors into the duration calculation, as [the guide on testing and consent](https://conversion.club/playbook/a-b-testing-and-consent-mode-what-do-your-tests-really-measure/) details. --- ## The line to present On launch day, send this sentence to your leadership. Copy it, then fill in your end date: > "Shopify splits the traffic and displays rates, without saying which group wins. We'll calculate the verdict ourselves on \[end date\], set before launch."