AB testing (also written A/B testing) is an experiment that shows 2 versions of a page, email or ad to randomly split visitors at the same time and compares one goal, such as orders. The random split means the gap comes from the change, not the season.
AB testing (also written A/B testing) is an experiment that shows 2 versions of a page, email or ad to randomly split visitors at the same time and compares one goal, such as orders. The random split means the gap comes from the change, not the season.
How does AB testing work?
You split traffic at random, show each group one version, and count the same goal for both until a sample size you set in advance is reached. Then you read the result once.
The split is usually even: Google Ads recommends 50% for custom experiments because it gives the best comparison. AB testing works best on one change at a time. Take a hypothetical candle store testing its button text, “Add to cart” against “Buy now”, the kind of small change Google says can have a surprising impact on what shoppers do.
After 1 week, version A has 118 orders from 4,200 sessions (2.81%) and B has 139 from 4,150 (3.35%). B looks better, but a statistical significance check gives p = 0.153, so chance explains the gap. After 3 weeks, 398 of 14,200 against 453 of 14,150 (2.8% vs 3.2%) gives p = 0.049, just under the usual 0.05 line.
Does split testing hurt SEO?
Not if you follow Google’s rules: show Googlebot what shoppers see, mark variant URLs as copies with a canonical tag, redirect with 302s, and end the test when it’s done.
Google’s website testing page treats showing one set of URLs to Googlebot and another to people as cloaking, which breaks its spam policies whether you’re testing or not. If each version has its own URL, put a canonical link (rel=canonical) on the variants pointing to the original; Google prefers that to a noindex tag here. Send visitors to a variant with a 302 (temporary) redirect, never a 301. On timing, Google says “a good testing tool tells you when you’ve gathered enough data to draw a reliable conclusion”, and after that you should remove the test elements as soon as possible.
Which testing tools do online stores use?
On 18 October 2026, 15 of the 28 store sites we read loaded a testing or personalization tool, and none still referenced Google Optimize.
We searched each homepage and the 17 Google Tag Manager containers on them for 14 tools. Dynamic Yield and Intelligems appeared on 4 stores each, Optimizely on 3, GrowthBook on 2, Monetate and AB Tasty on 1. Intelligems arrives as a Shopify app block on Everlane’s page. Google’s own tool is gone: “Google Optimize and Optimize 360 are no longer available as of September 30, 2023.” Google now points to third-party integrations with Google Analytics and names AB Tasty, Optimizely and VWO. The other 13 stores load none of the 14, which doesn’t prove they never test; server-side AB testing leaves no trace in the page.
Setting up a fair test
Good AB testing is mostly planning. Decide 3 things before launch: the one change, the goal, and the sample size. For the candle store, telling 2.8% from 3.2% took about 14,000 sessions per version, which a store with 2,000 sessions a day reaches in 2 weeks. A store with 300 sessions a day would need over 3 months, so test bigger changes first.
Measure orders or revenue per session rather than clicks on the tested button, and keep the conversion rate definition the same for both versions. Run whole weeks, so each version sees the same weekday mix. Don’t change prices, ads or the email schedule mid-test either; anything that hits one version harder than the other breaks the comparison.
Email is the easiest place to start AB testing on low traffic: a subject line test on a list of 20,000 splits into 2 groups of 10,000 in one send, where a product page with 300 sessions a day would take weeks to collect the same sample. Our conversion rate optimization service plans these numbers before any split testing goes live.