Native A/B Testing on Shopify: How Rollouts Works and What It Can't Do Yet

For years, the answer to "how do I A/B test on Shopify?" was some version of: pick a third-party app, pay the monthly fee, deal with the injected scripts slowing down your pages, and hope the methodology holds up. Shoplift, Intelligems, Convert, Optimizely, all capable tools, all adding cost and complexity to an already crowded tech stack.

That changes with Rollouts, native split testing built directly into the Shopify admin. Announced in the Winter '26 Edition, it's currently rolling out in early access.

What Rollouts Is

Rollouts is built directly into the Shopify admin. No app install required, no third-party scripts, no additional monthly cost on top of your existing Shopify subscription. You access it from Online Store > Themes, where a Rollouts button appears next to your published theme.

The core mechanic is clean: you create a rollout, give it a name, and set a traffic percentage, which is the share of visitors who will see the variant instead of your live theme. From there, you click through to the theme editor and make your changes. You can schedule the rollout to start and end at specific times, which is useful for promotional periods or seasonal updates where timing matters. In practice, the setup is smoother than most merchants expect. There's no configuration overhead, no script installation, no waiting for a support team to provision your account.

The result is a true split test running inside Shopify's own infrastructure, with results flowing into your standard Shopify analytics. No extra layer. No script conflicts. No worrying about whether your page speed scores are being dinged by a testing tool.

The Workflow Shopify Intends

Rollouts is designed to work in tandem with SimGym, Shopify's AI-powered storefront simulation tool, in a specific sequence:

1. You're planning a meaningful change (a new theme, a redesigned PDP, a restructured navigation)
2. You run a SimGym simulation first to get synthetic early signal. Does this direction cause confusion? Where do simulated shoppers drop off?
3. You make refinements based on those findings, then set up a Rollout at a conservative traffic percentage (10-25%) to test with real customers
4. You monitor performance, scale up traffic to the variant if results are positive, and let the winner take over

It's essentially a two-stage validation pipeline that Shopify has now built natively into the platform. The first stage uses AI to catch the obvious problems before real visitors are exposed. The second stage uses real traffic to confirm the direction with statistical confidence.

What Rollouts Can (and Can't) Test Right Now

This is where it's important to be precise, because Rollouts is genuinely useful for what it does while being more limited than a full-featured conversion rate optimization (CRO) platform. Understanding the scope upfront saves frustration.

What you can test:

• Full theme changes (swapping to a different theme variant)
• Section layouts and visual design changes made through the theme editor
• Navigation structure and menu changes
• Promotional banners and seasonal content
• Homepage layout and collection page organization

What you can't test yet:

• Global theme settings (colors, typography, and other settings configured outside the section editor)
• Checkout flows or checkout page elements
• Product pricing or discount structures
• App embeds and third-party app content
• Liquid template changes
• Audience-segmented variants (no "show version A to returning customers, version B to new visitors")
• Custom conversion goals beyond Shopify's standard analytics
• Deep reporting. The built-in analytics cover the basics, but early adopters have noted the reporting lacks the depth you'd get from a dedicated testing platform (no confidence intervals, no segmented breakdowns, limited export options)

Shopify has stated they plan to expand Rollouts to additional surfaces, including products, discounts, and other areas, over the coming months. The initial release is deliberately scoped to theme-level changes, which is the highest-risk area for most merchants and the most common source of conversion regressions after major updates.

What This Means for Shopify Plus Brands Specifically

The early access release is available across Shopify plans, but the full feature set is weighted toward Shopify Plus. That framing makes sense given the context: brands doing serious volume have the most to lose from a poorly validated redesign, and also have enough traffic to generate statistically meaningful test results in a reasonable timeframe.

If you're a Shopify Plus brand running 200,000+ monthly sessions, the economics of Rollouts are straightforward. You're currently spending somewhere between $300 and $1,000+ per month on third-party testing tools, plus the dev time to configure them correctly and the ongoing maintenance when a Shopify update breaks something in their script injection. Rollouts eliminates most of that stack for theme-level tests. You keep your third-party tools for the scenarios they're built for, like granular element testing, pricing experiments, and checkout optimization, and use Rollouts for the broader structural changes that were previously the most nerve-wracking to ship.

If you're a Shopify Plus brand running lower traffic, say 30,000 to 80,000 monthly sessions, Rollouts still has value, but you need to be disciplined about test duration and traffic allocation. A 50/50 split at lower volumes requires more time to reach statistical significance, and the temptation to call a winner early is the most common way testing goes wrong. The right move at lower traffic is often to run the test longer at a higher traffic percentage rather than trying to move quickly on thin data.

Rollouts vs. Third-Party Shopify A/B Testing Apps

The honest answer is that Rollouts doesn't replace Shoplift, Intelligems, or Convert for brands running sophisticated experimentation programs. Those tools can test individual page elements without duplicating an entire theme. They support audience segmentation. They have statistical rigor built into their interfaces: confidence intervals, minimum detectable effect calculators, the guardrails that prevent you from making bad decisions on early data.

What Rollouts does is something different: it removes the barrier to entry for brands that weren't running any formal testing at all. The merchant who was making design decisions based on gut feel, or launching a new theme on a prayer, now has a way to de-risk those decisions natively, without a monthly subscription, without dev work to configure an external tool. That's a meaningful shift for the broader Shopify ecosystem.

For brands that are already running a mature CRO program, Rollouts is a useful addition, particularly for the "we're pushing a big theme update and want to graduate it to 100% of traffic cautiously" use case. The percentage-based rollout logic is clean and well-suited to that workflow. Start at 10%, hold for a week, check your core metrics, scale to 25%, repeat. It's a safe deployment pattern that experienced teams will recognize immediately. Early adopters consistently point to this gradual graduation as the feature's biggest practical value. The ability to push a winner incrementally rather than flipping a switch feels meaningfully less risky than the traditional all-or-nothing theme publish.

Getting Started with Shopify A/B Testing

Rollouts is in early access now and accessible through the Themes section of your Shopify admin. Look for the Rollouts button next to your published theme. The interface is intuitive: create a rollout, set your traffic split, make your changes in the theme editor, schedule the deployment window, and monitor results in your Shopify analytics dashboard.

The most important thing you can do before your first rollout is document your baseline metrics clearly: conversion rate, add-to-cart rate, average order value, and bounce rate on the pages you're changing. The test is only as useful as the baseline you're comparing it against.

If you want to run the full SimGym + Rollouts workflow before your next major update, get in touch with the Conspire team. We've been working with both tools and can help you set up a testing process that actually generates reliable signal.