There are numerous benefits to testing in online marketing. However, testing isn’t everything. And it’s critical for marketers to remember what it is and what it isn’t.
Testing is a way to validate the ideas for conversion rate improvement that you deploy on your site. That’s it. Testing just tells you if you’ve made a good decision, but it doesn’t give you the ideas.
Aside from not treating website testing as if it’s synonymous to conversion rate optimization (CRO), here are 5 situations when you shouldn’t test:
1. When problems are obvious
Forget testing and just fix the glaring problems on your site. While it’s true that you should test most important things, you should still consider best practices on effective copywriting, making the call-to-action clearer, and removing visual distractions.
If best practices worked 9 out of 10 times, that’s probably higher than the success rate for most of your tests.
2. When problems are site-wide (not isolated)
Testing individual pages is okay, but don’t forget to look at the whole experience.
For instance, if you create alternative versions of your product detail pages that are better than what they replaced, the problem will be the rest of the site experience. You might have that horrid homepage with sliding banner ads, your products might be impossible to find, or your checkout process usability might be poor.
Your site’s performance will suffer because of the contrast between the part you fixed and the context around it that still leaves a lot to be desired.
A lot of times, things can’t be fixed by tactical testing because they’re not isolated to one part of your site – they’re part of the larger experience.
3. When there is not enough traffic
Even with basic A/B testing, if you don’t have 10 conversion actions a day, you shouldn’t be doing tests.
You can try to compensate for the data rate and measure different things. For example, the conversion action in an e-commerce scenario might not be an actual sale- it might be adds to cart. If you know, for instance, that 1/3 of things added to the cart are eventually bought (so you have 3 times as many adds to cart as you do sales), you can measure that if you’re fixing the upstream part of the experience.
The conversion action might also be just a navigation click. If you have the conversion action deeper in your site, the micro-conversion of a click in the right direction might be a good tactical independent thing to measure and test for.
4. When traffic is changing, seasonal, or spiky
If you can sample people from a certain population or group and figure out how they behave, then you’ll be able to predict the behavior of future visitors to your site.
The problem with the world of online marketing, however, is it’s not clean at all.
It’s difficult to sample from the same population – you have competitors, seasonality, and changes to your traffic mix. And anything fundamental that you change is going to affect the test, and you no longer get that purity of predicting how a certain group of people acts.
For example, if you have a tax offer, the people taking on the offer 6 months in advance are not the same as the ones ordering it a week before the tax deadline. For flower websites, customers care about urgent delivery the day before Mother’s Day. They don’t care about price like the people who plan ahead. In travel, you have seasonality with people from cold places coming to warm places in the winter, and vice-versa.
These are all different populations.
So, if you have strong seasonality or changing demographics or behaviors coming to your site, you can’t extrapolate that. If you do a test on a flower site a week before Mother’s Day, you can only use that information for that same group of people. You can’t use it in off-season because people will behave differently.
If you know that there’s some seasonality or a holiday when people act weird, quarantine that part of the data out when you’re doing your analysis.
5. When you are ignorant of basic math
Some people ignore the math and the statistics altogether and just try to lock in the winner right away. You really can’t get proper statistics without collecting enough samples. If you’re working with too little data, cutting tests off early, or not testing in full week increments, you’re not doing math.
You might as well be flipping a coin.
Keep Testing in Proper Perspective
Testing has its place in online marketing, but it’s not CRO. And there are situations when testing will do more harm than good. If the problem is conspicuous or site-wide, or if you don’t have the data rate or traffic is seasonal or spiky, you should skip testing. Sure, you can do some things to mitigate these problems, but a lot of times, you just can’t test because the signal is buried in the noise.
Look instead at whatever qualitative insights you can get from your visitors, other kinds of user research, or best practices.
