How to Fail at Landing Page Testing

Posted by | Comments

Tags: , , ,

how to fail at landing page testing

In my last column, I wrote about some of the ridiculous excuses we often hear from companies for not starting a conversion optimization program. You should obviously try to improve your conversion rate. But in many people's minds there is a mistaken perception that landing page testing is the only tool available. The fact is, there are many situations where testing is just plain wrong, and probably doomed to failure:

  • If your data rate is too low
  • If your website experiences seasonal spikes in traffic
  • If your website has major issues

If Your Data Rate Is Too Low

Say your site is fairly new and you're only seeing a few conversions per day. You're beginning to panic because you have a low conversion rate and your marketing person tells you that you should be testing so you can optimize for conversions. Unfortunately, this is one situation where your tests would most likely be inconclusive.

Testing is built on probability and statistics. So we recommend at least 10 conversion actions per day in order to conduct even a basic A/B split test. Below that level you will have too much "fuzz" and variability in your answer and will probably be unable to reach conclusive results. Or your test can stretch out over many months of data collection, leading to problems with seasonal changes in your audience's behavior.

One way to deal with a low data rate is to change what you consider to be the conversion action. If you don't have enough data to test the bottom-of-the funnel macro conversion (e.g., the sales in an e-commerce setting), go up the conversion path to find more granular and frequently appearing micro conversions, which can still ultimately impact the bottom line. For example, you can measure the number of adds-to-cart instead. These happen more frequently, and serve as a proxy for the actual down-funnel conversion that you want to measure (completed sales at the end of the checkout).

If Your Website Experiences Seasonal Spikes in Traffic

It's a month before Mother's Day, you're in the flower business, and your test-happy team wants to dust off one of those wish list items that's been sitting in the inbox for months, starting with an important landing page on your site. Just say "no"!

Testing is based on the assumption that you have a homogenous group coming to your site. If you can predict how a subset of that group acts, you can generalize that to how the rest of the users are going to act. But businesses that face seasonal shopping trends have a very different traffic mix during peak season.

For example, if you're in the flower business and a month out from Mother's Day new visitors to your site are likely price shopping, and that's the only criteria they care about: your price. These are the people who plan ahead (and probably buy their Christmas present in July as well).

Two days out from Mother's Day, the visitors who show up to your site are in a panic. These visitors are looking for express shipping and they do not care about paying extra for the flowers or the shipping. They just want to make sure that Mom gets them on time. These procrastinators (myself included) are part of a very different group.

The point is, visitors in the months leading up to the holiday are going to behave differently. If you tested for price sensitivity two months out, you'd see there was a lot of it; if you tested days out, you wouldn't see much because visitors care more about availability.

Very spiky seasonality complicates your tests. To adjust for this, you might tune your CRO for the short-term seasonal period - and only apply your findings within the same period the following year. Or you might conversely filter out any short-term peak-season impacts and treat them as anomalies, and tune for the baseline behavior throughout the rest of the off-peak year.

Here are some tips for dealing with seasonal business and testing:

  • Choose recurring, controllable, and stable traffic sources.
  • Don't change your marketing mix during the data collection.
  • Plan data collection around any major product or company announcements.
  • Average across obvious sampling biases (day-of-week, time-of-day).
  • Schedule around any known seasonal trends or important industry events.
  • Remove or mitigate data from unexpected external events.
  • Continue ongoing low-level data collection on the original recipe even after the test is completed.

If Your Website Has Major Issues

You've been testing and testing and not getting any conclusive results. You're trying to pinpoint the problem but can't seem to figure out what's going wrong. But the problem may be bigger than any one of your tests. The problem is your whole website experience.

If your site has major fundamental problems with its navigation, design, or usability, testing piece by piece is very inefficient because what the site really needs is to be rebuilt and redesigned from the ground up.

If that sounds extreme, think about the fact that conversions don't just happen in a bubble. You want your site to offer a seamless experience for users overall. If you have a crappy site, putting Band-Aids on landing pages here and there is not the answer.

When visitors are meandering around your site without clear purpose or direction, commonly using the back button, or relying on the search function over navigation, you probably have a website problem.

The overall goals of web usability are:

  • Decrease the time it takes for visitors to finish tasks.
  • Reduce the number of mistakes visitors are likely to make.
  • Shorten the visitor's learning time overall.
  • Improve the visitor's satisfaction.

Your site's major issues can be remedied by focusing on three key areas:

  1. The visual presentation of your site, including page layout, graphics, color, buttons, and the decision to use on-page animation or video.
  2. Information architecture, including content categories and how easy it is for visitors to understand how the site is organized so they can quickly find what they are looking for.
  3. The content and how it's written, making sure it's free of jargon, formatted for readers on the web, and provides the information visitors seek.

So the next time someone tells you that you need to test that, keep in mind not all environments are sufficiently testable. If you work to understand the big picture, then you can often find alternatives to test that lead to a shared goal, or you can figure out how to best deal with data that can throw your results.


This article originally appeared in Tim's ClickZ column July 23, 2013

'Want to optimize your conversion funnel but don't know where to start? Find out how we can help you with conversion rate optimization