There are generally two reasons marketers conduct split tests. The first one is continual site improvements – getting more conversions, lowering the bounce rate, and so on. The second reason is to get data so that you can tell Randy from management to “shut up about your rotating carousels already.”
As a marketer, you generally want to focus your tests on continual site improvements. (sigh) Management rebuttal is required every now and again (and let’s face it – fun!) when you’re trying to avert disaster, but over several cycles, it’s less productive than testing for site gains.
So if you’re testing for the right (and less fun) reasons, there are generally things you can skip. After all, your conversion lifts are only going to be as good as your hypotheses for your site. You can test these when they are unavoidable, but you generally want to stay very far away from these ideas.
Rotating Banners
So … start with the obvious, right?
A lot of landing pages have rotating carousels, and there’s been significant testing devoted to this.
Pages with rotating banners suffer from banner blindness : users generally ignore things that look like ads, so they are more likely to skip rotating carousels altogether. Even for the small number who pay attention, though, there’s usually not enough time or control to get through the content before the banner auto-rotates.
If you absolutely have to test them because it’s a mandate, there are ways to make rotating banners less … deadly. But if you can, the best idea is really to stay away.
Broad Intent, Deep Page
All right, this one is generally for SEOs who are trying to hit a large set of search terms using one page, or site creators or designers trying to make a deep page fulfill several tasks.
At the top layer of your site, it’s okay to have a page cater to a larger set of tasks. From the top layers of your site, it’s okay to have research, price-comparison, purchasing, and educational tasks all laid out as options. As a user gets deeper into the web site, the idea is that they will go from broad to specific – and the deeper you get into the site, the more specific the intent is that the page caters to.
Every now and again, you see product or service pages that violate this. Those pages try to do everything in one go. The result is that you have a ton of deep-layer content, a slew of calls-to-action, and not enough attention being drawn to the key task of the page.
If you’re testing pages like these, you’re starting with a handicap. You generally want to test pages higher up in the information architecture for broad tasks, or deeper into the information architecture for specific tasks – don’t mix and match.
Long Mobile Forms
While there’s something to be said for the greedy marketer who is pinpointing the exact amount of pain a user is willing to tolerate on a mobile device before he or she quits, you generally want to steer clear of this tactic.
If something should not require a form, give it away without the form. If something requires a form on a mobile device, make sure it’s pretty short. Even when it’s pretty short, you should help your users out and provide the option for auto-complete.
After you have all of that down, then you can split test for titles, the absolute required fields, and so on. But you have to get the basics right, first.
Split Testing For Wins
You should be testing a lot of ideas. Still, starting with things that have either repeatedly shown to lose in tests, like rotating carousels, or those that are just plain painful for users, like long forms, is bad for time – for the most part.
That having been said, when you absolutely need to, it’s a blast setting up a/b tests so you can say “Randy, enough with the form fields already.”
