Native A/B Testing on Shopify: How Rollouts Works and What It Can Do Now

Rollouts is built directly into the Shopify admin. No app install required, no third-party scripts, no additional monthly cost on top of your existing Shopify subscription. You'll find it under Markets > Rollouts.

The core mechanic is still clean: you create a rollout, choose what it changes, and set a traffic percentage, which is the share of visitors who see the new version instead of your live store. Since September 2026, setup starts by asking what you're trying to do, and the settings adjust to match:

• A launch: schedule a change to go live at a set date and time, or publish it gradually to a growing share of traffic.
• A temporary event: swap in a different setup for a fixed window, like a BFCM theme for one week, and the store reverts automatically when it ends.
• An experiment: split traffic between versions, review the impact on conversions, and pick the winner.

When you start a rollout, Shopify makes a copy of your published theme or configuration, so your team can keep editing the live store while the test runs. The setup screen shows how much traffic the rollout actually reaches and how it's split between versions, and it flags conflicting settings before you schedule anything.

The result is a true split test running inside Shopify's own infrastructure, with results flowing into your standard Shopify analytics. No extra layer. No script conflicts. No worrying about whether your page speed scores are being dinged by a testing tool.

• March 2026: generally available on Basic plans and above, with scheduling, automatic revert after events, A/B tests against your current theme, and market targeting (on Advanced and Plus). Theme settings and app embeds can also differ per market or per rollout, so colors, typography, app embeds and custom CSS are testable now.
• June 2026: checkout and customer account configurations joined themes. You can schedule them, roll them out gradually, or A/B test two entirely different setups, and run several experiments at the same time without them overlapping.
• September 2026: the intent-based setup above, plus combining theme, checkout and customer account changes in a single rollout.
• October 2026: discounts. You can launch an offer to part of your online store traffic, compare it against your existing offer, and ship it alongside theme and checkout changes in the same rollout. Apps can now read rollouts through Shopify's API too, which is what lets third-party tools and custom integrations respect a test instead of fighting it.

Rollouts is designed to work in tandem with SimGym, Shopify's AI-powered storefront simulation tool, in a specific sequence:

1. You're planning a meaningful change (a new theme, a redesigned PDP, a reworked checkout, a new offer)
2. You run a SimGym simulation first to get synthetic early signal. Does this direction cause confusion? Where do simulated shoppers drop off?
3. You make refinements based on those findings, then set up a Rollout at a conservative traffic percentage (10-25%) to test with real customers
4. You monitor performance, scale up traffic to the variant if results are positive, and let the winner take over

It's essentially a two-stage validation pipeline that Shopify has now built natively into the platform. The first stage uses AI to catch the obvious problems before real visitors are exposed. The second stage uses real traffic to confirm the direction.

This is still where it's important to be precise. Rollouts covers far more than it did at launch, but it is not yet a full-featured conversion rate optimization (CRO) platform.

What you can test:

• Themes: a full theme against another (including code-level changes, since the variant can be an entirely different theme), section layouts, navigation, homepage and collection layouts, promotional content
• Global theme settings and app embeds (colors, typography, custom CSS), per market or per rollout
• Checkout configurations, such as a new upsell block on the checkout page
• Customer account pages, such as a new branding experience
• Discounts and offers, on online store traffic
• Different versions per market, like a localized homepage for European buyers (Advanced and Plus)

What you can't test yet:

• Product prices directly. Discounts are covered, and Shopify's API already lists catalogs (price lists) as something a rollout can change, so price testing looks close, but it isn't in the admin announcements yet
• Audience-segmented variants beyond market (no "show version A to returning customers, version B to new visitors")
• Custom conversion goals beyond Shopify's standard analytics
• Deep reporting. The built-in analytics cover the basics, but the reporting still lacks the depth you'd get from a dedicated testing platform (no confidence intervals, no segmented breakdowns, limited export options)
• Partial-traffic discount tests outside the online store. A discount at less than 100% of traffic only applies on the Online Store channel

On September 21, 2026, Shopify changed how Analytics counts sessions: more consistent bot filtering, and shopper journeys that don't include a standard pageview now count. Sessions, conversion rate, add-to-cart rate and checkout rates can all move even though orders and sales didn't.

That matters for Rollouts because its results are read through those same session-based metrics. Don't compare a test that started after September 21 against a conversion baseline from before it, and if a rollout spans that date, treat the before and after periods separately. Orders and revenue per visitor are the safer numbers to lean on across the change.

Rollouts is available on Basic and above, and market targeting needs Advanced or Plus. The brands that get the most from it are still the ones doing serious volume: they have the most to lose from a poorly validated redesign or checkout change, and enough traffic to reach a reliable result in a reasonable timeframe.

If you're a Shopify Plus brand running 200,000+ monthly sessions, the economics are now hard to ignore. You're currently spending somewhere between $300 and $1,000+ per month on third-party testing tools, plus the dev time to configure them correctly and the ongoing maintenance when a Shopify update breaks something in their script injection. Rollouts now covers theme, checkout, customer account and discount tests natively, which is most of what that stack was doing for many brands. Keep a dedicated tool where you need per-element tests, audience segmentation or deeper statistics.

If you're a Shopify Plus brand running lower traffic, say 30,000 to 80,000 monthly sessions, Rollouts still has value, but you need to be disciplined about test duration and traffic allocation. A 50/50 split at lower volumes requires more time to reach statistical significance, and the temptation to call a winner early is the most common way testing goes wrong. The right move at lower traffic is often to run the test longer at a higher traffic percentage rather than trying to move quickly on thin data.

The gap has narrowed a lot since February. Checkout and offer testing, two of the main reasons brands kept a third-party tool, are now native. What Shoplift, Intelligems and Convert still do better: testing an individual page element without duplicating a whole theme, segmenting by audience, and building statistical guardrails into the interface, like confidence intervals and minimum detectable effect calculators that stop you from deciding on early data.

For brands that weren't running any formal testing at all, Rollouts removes the barrier completely. The merchant who was making design decisions based on gut feel, or launching a new theme on a prayer, now has a native way to de-risk those decisions, without a monthly subscription or dev work to configure an external tool.

For brands already running a mature CRO program, Rollouts has gone from a useful addition to the default place to stage big changes. The coordinated launch is the standout: a new theme, the checkout changes that go with it and the launch offer can ship together in one rollout, at 10% of traffic first. Start at 10%, hold for a week, check your core metrics, scale to 25%, repeat. The ability to push a winner incrementally rather than flipping a switch is still the feature's biggest practical value, and it now applies to checkout and discounts as well as the theme.

Go to Markets > Rollouts in your Shopify admin and create a rollout. Pick whether it's a launch, an event or an experiment, choose what it changes (theme, checkout, customer accounts, discounts), set your traffic split and schedule, review any conflicts the setup flags, and monitor results in your Shopify analytics.

The most important thing you can do before your first rollout is still to document your baseline metrics: conversion rate, add-to-cart rate, average order value, and bounce rate on the pages you're changing, measured after September 21 so the sessions change doesn't skew the comparison. The test is only as useful as the baseline you're comparing it against.

If you want to run the full SimGym + Rollouts workflow before your next major update, get in touch with the Conspire team. We've been working with both tools and can help you set up a testing process that actually generates reliable signal.