Test That Carefully: Avoid the Biggest Testing Mistakes

Posted by | Comments

Tags: ,

avoid the biggest testing mistake

You know what you don’t hear a lot from digital marketers and CROs? They typically don’t say “You should be running fewer tests,” or even, “You’re testing THAT?” Nope - most of the time, it’s all, “You should test that.”

Part of that’s because for years now, testing has been the most neglected critical activity in the space. It’s easy for C-level executives to see the value of a top-ranked search term in Google - ensuring visitors have a clear line of sight to the purchase, not so much. So there’s the evangelism piece to this, and then there’s the ongoing reaction to things like truly awful reasons companies don’t have a CRO budget.

But then there’s the other reality: meaningful testing is difficult. It’s not quantum physics-hard, but it is hard. Anyone who can edit header tags and change a button color or two can run a split test, but meaningful tests require earned specialization, incurred expenses, allocated resources, and a functional framework for web sites. You can test and lose money, so it’s very important to be deliberate in how you test.

All that said, testing may be difficult, but it is critical. So if you’re going to run tests, you might as well get it right, and avoid the worst mistakes out there.

1.) I have a plan: attack 

One of the really cool things about testing is that it takes the decision-making away from management opinions and it focuses it on cold, hard, data. 

Or does it?

Because if you’re split testing a page with fewer than 10 conversions per day, or a micro-site with insignificant traffic, you’re taking decisions away from opinions -- and driving it into crazy, random happenstance. Tests require a significant amount of traffic and conversions to be conclusive.

Or even if your page does have significant traffic and conversions, if you’re testing it close to Cyber Monday, the traffic mix is about to change, and that will render your results unreliable. Make sure you have a carefully thought-out plan of attack.

How to avoid this: Make sure that you have enough stable, recurring, controllable traffic, or say no to the test. (Rule of thumb: test pages with at least 10 conversions per day, and avoid Christmas, Valentine’s Day, and other visitor behavior-altering events)

2.) Red pill or blue pill 

When marketers pitch testing budgets to management, there’s a need to simplify what makes testing so critical, and some of the nuance is lost. The pitch can sometimes degenerate into finding out whether the red or the blue button works better - and that’s really not what optimal tests are about.

Some elements do indeed have a close relationship to the conversion, and every now and again you might indeed strike gold and find something that can be generalized for the site.

Most of the time, though, that kind of test is a waste of time and resources. You should test whether a search box helps or impedes visitors from finding what they need. You should test versions of a high-impact page as it’s overhauled.

How to avoid this: You should refine into small, granular tests when the visitor intent is already being served well, the page already has the optimal number of fields and elements, the wording is already precise, and all of those have been tested and optimized.

3.) I’d like to test that theory 

This is related to red-pill-vs-blue-pill, but one of the key testing mistakes is testing “elements” before you have a theory about the behavior you are trying to encourage.

Before you have a test, you should have already done the research. You should already know that there’s an issue somewhere, you should have a theory about what’s wrong or what can be optimized better, and you should have a solution in mind before you go to management and say, “I’d like to test that theory.”

How to avoid this: If you don’t have a compelling reason to test something, don’t test it. Run the research, trust the process, find the biggest user-experience gaps, and test against those instead.

All it needs is a little push

Testing is a critical tool in your arsenal, but it’s easy to get it wrong and do more harm than good. You can’t afford not to do it - all your competitors are refining their web presence. All you can really do is come prepared - find the best practices, avoid the worst practices above, and keep learning about the space as you test away. It gets easier to find the holes to test over time, and the process does get more efficient - the key thing is to be aware of the potential missteps, get started, and keep refining the testing process as your testing maturity level rises.


'Want to optimize your conversion funnel but don't know where to start? Find out how we can help you with conversion rate optimization