Podcast: Testing Screwups and Igloo Building with Joel Harvey

Posted by | Comments

Tags: ,


In this episode of Landing Page Optimization, Tim Ash chats with Joel Harvey, managing partner at Conversion Sciences, about conversion tests, common testing mistakes and how smart businesses can avoid making them.

Joel explains that Conversion Sciences focuses on uncovering things that can be tested on their clients’ web sites and executing the tests. He says they make the clients the most incremental value this way because they’re assured that the clients don’t make the common testing mistakes. 

Placing the Wrong Bets: Poorly Designed Tests 

The first thing that can make a test go wrong, according to Joel, is a bad hypothesis or a hypothesis that doesn’t isolate the right variables or any variables, as this produces bad data that’s going to lead you to making bad decisions. 

It’s a mistake to change the call-to-action button, the headline, and the image, all in one go, for instance, because even if there’s a conversion lift, it would be tough to associate it to a specific change.

Opinion-Driven Organizations: “If that wins, we’d never put that on our site!”

Joel says the biggest mistake companies make is being not data-driven; this is synonymous to not being driven by the growth that can be achieved through optimization. 

He shares that ideas are often met with resistance because of branding concerns. Tim points out that people, generally, don’t notice when something’s slightly off-brand, and that brand guardians have to see that, through testing, they get information from real customers. 

Test Implementation: Getting through IT and Cross-Browser Testing

Joel and Tim then tackle how to get through the IT barrier. Typically, tests run by having a small piece of javascript introduced into the web site, and just as marketers protect the brand against inconsistencies, IT protects the systems from risks. Joel indicates that showing a list of web sites deploying the codes successfully usually convinces companies to deploy the codes.

There’s also the matter of QA, and Joel notes that QA is really one of the things that you pay for. He notes that everybody might like sausage, but no one wants to go to the sausage factory, and in online tests, QA is that sausage factory. You can have the greatest ideas in the world but if you roll them out and they’re not working for important segments of your user base, then it’s not going to matter.

So it’s important to do cross-browser testing, use different devices or use previews of devices to see if there are broken experiences, or virtual machines for different operating systems.

Avoiding Common Mistakes

Joel and Tim note that you shouldn’t wait for a brilliant idea to start testing. The important thing is to try and understand what works and what doesn’t, and then start removing some of the things that don’t work for tests. It’s usually not about changing the elements on the site, but removing things that are on the site – too many things are usually going on.

The other thing to note is that there’s a local maxima – you can optimize an existing page only so much before you’ll hit a wall. Still, Joel notes that you can probably double your conversion rate before hitting that, so you can do the easy part first before trying something more drastic, like redesigning your site.

First Air Date: February 3, 2014

Listen online or download the podcast from WebmasterRadio.fm