Nearly every Shopify A/B testing roundup still opens the same way: Shopify has no native A/B testing, so you'll need an app.

As of September 1, 2026, that's wrong.

A/B testing is in the Shopify admin under Markets > Rollouts. The help center defines it plainly:

An experiment is a type of rollout for testing a new strategy by comparing control and treatment groups to evaluate the effectiveness of changes.

Per Shopify's changelog, rollout experiments shipped March 31, 2026, and expanded on June 5, 2026 to cover swapping entire themes and checkout configurations. It isn't Plus-only.

But this doesn't make paid apps obsolete. Experiments run into three hard walls: the plan requirement, the extremely narrow set of testable resources, and a reporting design where switching from a launch to an experiment removes gross sales and average order value from the results screen.

Below: exactly where Rollouts stops, why Shopify's own performance docs put a number on the cost of app-based testing, and how ten real apps price the gap. Research date: September 1, 2026. All app prices are USD as listed on the Shopify App Store.

The short version

If you're comparing a theme redesign against your current theme, Rollouts is enough. On Grow or above it costs nothing extra.

If you want to test one button, one image, one headline, Rollouts can't do it. The unit of an experiment is your entire main theme or your entire checkout configuration.

If you need to judge the winner on revenue, AOV, or profit, Rollouts won't show you. A theme experiment reports four metrics — conversion rate, bounce rate, reached checkout rate, add to cart rate — and the help center states you can't customize them.

If you want to test price, price isn't a rollout resource at all. Apps handle it, and almost all of them gate it behind a top tier.

If you are installing something, by scenario:

• Under 100 orders/month → Visually A/B Testing & CRO ($15/mo, priced on orders)
• Predictable cost at 10,000 monthly visitors → Personizely: A/B Testing & CRO ($39/mo, priced on unique visitors)
• Price, shipping, and checkout testing in one tool → Intelligems: A/B Testing (price testing lives on the $349 Unlimited tier) or ABConvert: A/B Testing (price testing from the $199 Growth tier; Built for Shopify)
• Built for Shopify badge on a mid-market budget → Shogun ‑ AB Testing ($39/mo, but only 3 concurrent tests)

And if any guide still tells you to use Google Optimize: it shut down on September 30, 2023. Fittingly, Shopify's developer docs still cite googleoptimize.com as a domain to look for when debugging a slow store.

What Rollouts actually does

Experiments require Grow or above

The plan requirements are split across two lines:

Rollouts are available on the Basic plan or higher. Experiments are available to stores on the Grow plan or higher.

On Basic you get launches only. A launch is defined as "a type of rollout with no change-level traffic splits" — it schedules changes, ramps them to a percentage of visitors, and can auto-revert at an end date. Useful for a seasonal campaign. Useless for comparing A against B.

So a Basic-plan store that wants a real A/B test has two options: upgrade, or install an app.

Only two resources are testable

The help center table is short enough to reproduce in full:

Resource	Change	Supports content changes
Online store theme	Edit main theme	No
Online store theme	Replace main theme	Yes
Checkout and accounts	Replace configuration	Yes

And: "You can add only one of each of the following types of changes to a rollout."

There is no concept of an element-level test. Changing a shipping-threshold bar or moving the review stars still runs as a whole-theme experiment. That works when you genuinely change one thing. Bundle three improvements together and you can't tell which one moved the number.

You don't have to duplicate the theme by hand, though. The help center lists "draft customizations directly on your main theme instead of creating a duplicate theme" as a rollout use case, and the June 5 changelog notes that a copy of your published theme or config is created automatically so you can keep editing the live one independently.

Three exclusions are stated outright:

• You can't change Liquid templates as part of a rollout
• Headless and custom storefront checkouts aren't supported
• Vintage themes aren't supported

Turning a launch into an experiment deletes your revenue metrics

This is the part most people will discover the hard way. The same theme change reports completely different metrics depending on rollout type:

Metric	Launch	Experiment
Gross sales	Yes	—
Average order value	Yes	—
Orders	Yes	—
Total sales over time	Yes	—
Sessions over time	Yes	—
First-time vs returning customers	Yes	—
Conversion rate breakdown	Yes	—
Conversion rate over time	Yes	Yes
Bounce rate	Yes	Yes
Reached checkout rate	—	Yes
Add to cart rate	—	Yes

A theme experiment gives you the four in the right column. A checkout experiment gives you exactly one: checkout conversion rate. And per the help center, "You can't customize the available metrics for your rollout."

Conversion rate up, AOV down, revenue flat is one of the most common outcomes in ecommerce testing. Rollouts experiments don't put the numbers you'd need to catch that on the screen. That single design decision is most of the remaining business case for paid apps, which is why they market on revenue per visitor, profit, and COGS.

The same page adds a warning worth internalizing:

If you run the same set of changes as a launch rollout and an experiment, then the results for the same metrics might differ. This is because each rollout type uses a slightly different algorithm.

Shopify doesn't publish the statistical method behind experiment results — Bayesian or frequentist, what threshold declares a winner, whether it corrects for peeking. That part is a black box.

Launch reach and split percentage multiply

Traffic allocation compounds. The help center's own example:

If you set the launch reach to 50%, and set the percentage of eligible visitors to 50%, then the percentage of eligible visitors for the experiment is 50% of the launch reach's total eligible visitors.

50% reach × 50% split means 25% of your traffic is actually in the test. The more cautiously you ramp, the longer you wait for a verdict.

Why app-based A/B testing slows your store

Shopify's developer docs have a dedicated page titled Remove A/B test anti-flicker snippets when no tests are running, and it attaches a number to the cost:

This performance cost is often over 2 seconds on First Contentful Paint (FCP) and directly hurts Largest Contentful Paint (LCP).

The same page distinguishes two things most articles blur together. FOOC — Flash of Original Content — is the control page appearing for an instant before the test script swaps in the variant. To prevent it, client-side testing tools ship a synchronous script that hides the page until the testing library loads. That hidden interval is the delay.

Shopify frames it as an explicit trade:

The trade-off: performance cost (2 or more seconds of FCP is common) against test accuracy (anti-flicker prevents FOOC) against the business value of insights.

And names the way out:

Server-side A/B testing has no anti-flicker performance cost. Content arrives pre-rendered with the correct variant.

Rollouts adds no anti-flicker snippet to your storefront. Shopify determines which eligible visitors see which variant, so the two-second cost above shouldn't apply. (Shopify doesn't document Rollouts' internal delivery mechanism, so treat this as an inference from the architecture rather than a documented guarantee.) The strongest argument for trying Rollouts first isn't that it's free. It's that it doesn't put rendering-blocking JavaScript on your critical path.

This also gives you a real question to ask vendors. Shoplift ‑ CRO & A/B Testing advertises "zero page flicker"; Visually A/B Testing & CRO advertises an "AB test-focused architecture." Ask what specifically that means: no anti-flicker snippet at all, or a faster one.

Price testing works differently than you'd think

Price isn't a rollout resource, and Shopify gives you no native way to show two prices for the same product in the same market at the same time.

So how do the apps do it? The permission lists give it away. ABConvert: A/B Testing requests "Edit Shopify Functions: Cart transforms, delivery customizations, payment customizations" plus "Edit discounts." Shogun ‑ AB Testing requests "Edit Shopify Functions: Cart transforms."

Price testing on Shopify is a display-price rewrite reconciled at the cart by a Function and a discount. Nobody is duplicating products into two URLs, so you don't fragment reviews or backlinks. But three layers — theme, cart, checkout — are now under app control.

That weight shows up in pricing. Price testing unlocks at Personizely Premium ($59), Elevate A/B Testing Price Test Standard ($99), ABConvert Growth ($199), Intelligems Unlimited ($349), Shoplift Advanced ($399, listed as "Price testing (beta)"), and Shogun Unlimited ($499). If price testing is the goal, compare the tier where it unlocks, not the entry tier — that ordering looks nothing like the entry-price ordering.

Ten A/B testing apps compared

Values pulled from each app's Shopify App Store listing on September 1, 2026. Review counts move daily.

App	Free plan	From (USD/mo)	Top tier	Rating (reviews)	BFS	Billing denominator
Theme Scientist A/B Testing	No	9.99	29.99	3.5 (10)	No	Monthly orders
Visually A/B Testing & CRO	Dev stores only	15	80	4.7 (124)	No	Monthly orders
Tenpla ‑ A/B Testing for Japan	No	19	39	0.0 (0)	No	Concurrent tests
Personizely: A/B Testing & CRO	No	39	149	5.0 (86)	No	Monthly unique visitors
Shogun ‑ AB Testing	No	39	499	4.3 (42)	Yes	Concurrent tests
Elevate A/B Testing Price Test	No	49	499	4.9 (130)	No	Order volume
Nelio A/B Testing	Yes (500 PV)	55	449	0.0 (0)	No	Tested page views
Intelligems: A/B Testing	No	69	349	4.7 (164)	No	Order volume
Shoplift ‑ CRO & A/B Testing	No	99	999	4.9 (120)	No	Monthly unique visitors
ABConvert: A/B Testing	No	99	599	4.7 (72)	Yes	Tested orders

Five different denominators means list prices aren't comparable

The right-hand column is the one that decides your bill. Monthly unique visitors, monthly orders, tested orders, tested page views, and concurrent tests are all in use.

A store doing 30,000 monthly visitors and 400 orders blows past Personizely's 10,000-visitor tiers and lands on Premium Plus at $149, while Visually covers it at $80. A store doing 50,000 visitors and 80 orders flips the ranking. You can't rank these apps until you substitute your own visitor and order counts — which is precisely the step most comparison posts skip.

Shoplift, Intelligems, and Elevate all note that pricing scales with sitewide traffic or order volume. Shoplift's Core plan displays "$99" alongside "Starting at $74," meaning the displayed figure is a ceiling, not a quote.

Uninstalling doesn't necessarily remove the code

A July 2026 one-star review of Shogun ‑ AB Testing documents, in unusual detail, that after uninstalling the app a snippet named shogun-optimizer.liquid remained in the theme, app-owned product metafields persisted, and displayed prices kept changing as a result.

The vendor's public reply doesn't dispute the mechanism:

Once the app is uninstalled, Shopify immediately invalidates our API tokens, which means we're no longer able to modify the theme or remove files as part of a cleanup process.

They add that the optimizer stands down after roughly 24 hours without a config refresh, and that the cart transform Function is removed by Shopify automatically on uninstall.

This isn't a Shogun-specific defect — it's the shape of every app that writes into your theme. Shopify says as much in its own performance guidance: "Uninstalling an app doesn't automatically remove its code from your theme." If an anti-flicker snippet outlives the test it was installed for, that 2-second FCP cost becomes permanent.

Is your store big enough to test at all?

One number to check before spending anything.

At 5% significance (two-tailed) and 80% power, here's the traffic a test needs. These are calculated from the standard two-proportion formula, not quoted from a vendor page:

Baseline CVR	Relative lift to detect	Sessions needed per variant
1%	20%	~42,700
2%	20%	~21,100
2%	30%	~9,800
3%	20%	~13,900
3%	30%	~6,500

A 2% CVR store trying to detect 2.0% → 2.4% needs roughly 21,100 sessions per variant — about 42,200 total. At 10,000 sessions a month, that's over four months, during which seasonality, ad spend, and inventory all change. It isn't a valid test anymore. Ramp cautiously in Rollouts and it takes longer still.

Below that threshold, the right move isn't a cheaper A/B testing app. It's qualitative: session replays, heatmaps, and reading what customers actually write to you. Microsoft Clarity: AI Insights is free and does the first two. Check your monthly session count in Shopify analytics before you check anyone's pricing page.

The short answer

As of September 1, 2026, the starting point for Shopify A/B testing is Markets > Rollouts. On Grow and above it's included, and because it's server-side it doesn't cost you rendering time. It fits the decisions where being wrong is most expensive: a theme rebuild, a checkout reconfiguration.

You graduate to an app when you need element-level tests, price or shipping tests, revenue-based decisions, or fine-grained segments. At that point, compare the billing denominator and the tier where price testing unlocks — not the number on the front of the listing.

And if your store doesn't clear roughly 10,000 sessions a month, no tool in this comparison will produce a statistically meaningful answer. Confirming that first is the cheapest optimization available.

---

All figures verified September 1, 2026 against the Shopify Help Center, Shopify developer documentation, and Shopify App Store listings. App pricing, ratings, and review counts change — check the current listing before installing.