4 Tips for Effective A/B Testing

A/B Testing is a learning process.

Kyle Rush, Head of Optimization at Optimizely, shares that when he started testing, he didn’t know anything about it.

He says it can be difficult to start your first test and figure out where to go from there, but that marketers should just get into it and learn as they go along.

This is not to say, though, that you should just do things randomly.

1. Start with the low-hanging fruit

Start with things that you can change within the marketing organization, without IT support, to build up some momentum for testing. This will get you on the path to getting buy-in for the testing program, then you can eventually get IT involved.

2. Control for time

A lot of things affect conversion rate, and time is the biggest variable. Test different versions at the same time, so the traffic that goes to each version is the same.

For example, if your PPC spending is increased while you’re testing version A, your audience will change dramatically. Then, if your PPC spending is reduced while you’re testing version B, your traffic won’t be as high quality.

So, don’t test sequentially.

Always split your traffic randomly and show different visitors different versions to get something more solid and stable.

3. Calculate your sample size beforehand

Optimizely’s free sample size calculator tells you the effect that you can detect reliably based on how long you want the experiment to run.

This is important to a successful testing program because if you are looking for a 1% increase in conversion rate, the calculator will probably tell you that you need a year’s worth of traffic.

If you’re a small startup company who doesn’t process much volume, a 1% increase isn’t worthwhile. You should be looking for bigger changes in your conversion rate instead. However, if you’re on a high-volume e-commerce site, a 2% increase in your “Add to Cart” rate makes a great difference.

Kyle shares that a company’s conversion rate is typically around 4%-10%, and that it’s difficult to pick up on differences that are below 5% or 10% because the baseline conversion rate is so low. So anything that’s very low impact, you will have to run for a very long time.

4. Run tests in increments of a week

If you run your test on a weekday but not on a weekend, you’re not going to get an accurate snapshot of your traffic because weekend traffic tends to behave differently from weekday traffic.

You can even see differences on weekday traffic (e.g. Monday traffic might respond differently from Friday traffic), so make sure you get a least a full week’s worth of traffic.

You’ll need to average across day of week or time of day, unless you’re specifically looking for the best weekend version and the best weekday during particular times of the day.