Tests that give an answer you can act on.

Worked example, with illustrative figures for a hypothetical store (not a client result): split testing whether delivery dates on product pages lift add to cart runs 50/50 for 3 full weeks with 26,000 visitors per variant, sized to detect a 10% relative lift from a 6% baseline. If add to cart moves from 6.0% to 6.4%, short of 95% confidence, the decision is to keep the original and test a stronger version.

We test our own work the same way. When we changed the tracking script on this site, we didn’t trust a single speed score: with several builds sharing one server, scores were noise. So we loaded the old and the new script on the same page, side by side, and compared them under the same conditions. Split testing a store follows the same rule: compare like with like, at the same time, and don’t read a result from noise.

Split testing starts with research, not ideas. We read your analytics funnel, session recordings and reviews, and shop the store on a phone, to find where people hesitate. Each finding becomes a hypothesis with a reason behind it, such as “shoppers can’t see the delivery date on phones, so adding it above the button should lift add to cart”. Nielsen Norman Group’s A/B testing guide makes the same point: tests work best when qualitative research tells you what to change.

Then we size the test before it starts. Sample size depends on three numbers: your baseline conversion rate, the smallest effect worth detecting, and the significance threshold, usually 95%. NN/g’s example is a 3% baseline click rate with a 20% minimum detectable effect. We run the numbers for your pages and turn them into weeks. If a product page would need four months to reach an answer, we say so and pick a page with more traffic or a bigger change.

Duration has a floor as well as a ceiling. NN/g recommends running a test for at least one to two weeks even with enough traffic, to cover normal swings in behaviour, and we always run whole weeks so a weekend sale doesn’t tilt one side. We don’t look at the result daily and stop at the first good day; the stop date is set before launch.

Setup must not hurt SEO. Google’s page on website testing sets four rules we follow: don’t cloak (show Googlebot the same variants as people), use rel="canonical" on variant URLs pointing to the original, use 302 rather than 301 redirects for tests that send people to another URL, and run the test only as long as needed. We also check that the variant doesn’t flash the old version before switching, which skews results on slow phones.

Reading the result is where honesty matters. A test that doesn’t reach significance is an answer too: the change didn’t make a difference big enough to detect, so you keep the simpler version. We report the conversion rate for each variant with its range, the revenue per visitor, and the result for phone and desktop separately, then write down the decision and what the next test should be.

When traffic is too low to test, we don’t pretend. Small stores get the clear fixes from research shipped directly, measured before and after, and only the riskiest changes, such as a new price display or checkout flow, go into a longer test. Our CRO audit covers that research step on its own.