Taguchi Sucks for Landing Page Testing

Posted by Tim Ash | Comments


I recently spoke on the multivariate testing panel at eMetrics in San Francisco. You would think that I dropped a hand grenade into the room when I opined that the Taguchi Method was a bad fit for landing page testing. This is a well understood fact to anyone with a solid understanding of basic statistics. Unfortunately this seems to leave out most landing page testers...

In the world of landing page testing there are two common mathematical approaches: A-B Split testing, and parametric Multivariate testing. A subset of Multivariate testing is known as "Design of Experiments" (DoE) and is also called "fractional factorial". A common fractional factorial approach is called the "Taguchi Method".

Some online marketers consider A-B Split testing to be kind of wimpy, and endow fractional factorial methods with an almost mythical quality.

I spend way too much of my time explaining to people that at least when applied to landing page optimization fractional factorial methods are a really bad idea. Despite this, the illusion persists that this kind of testing is somehow state-of-the-art, when in fact, nothing could be further from the truth.

Testing is composed of two important activities:

- Deciding what to test and coming up with good ideas

- Finding the best solution among your tested alternatives

People claim to get really good results with fractional factorial multivariate testing, and they credit this to the method that they use to analyze the data.

In reality, the improved conversion rates are the results of the great ideas for new landing page elements that go into the test. If all of your alternative landing pages designs are better then the original, it does not really matter what method you use to confirm that. Fractional factorial approaches may actually miss the best version of the landing page in your test and often lead you to a sub-optimal answer.

There is a huge mismatch between the original environment in which fractional factorial testing was developed and how it is usually applied to landing page optimization. It was basically transplanted to online marketing because it is relatively easy for a non-mathematical audience to understand, and not because of its appropriateness or fitness for the task.

The principal drawbacks of fractional factorial methods are:

  • Very small test sizes
  • Restrictive & inflexible test designs
  • Less accurate estimation of individual variable contributions
  • Drawing the wrong conclusions
  • Inability to consider context and variable interactions

Despite misinformation to the contrary, fractional factorial methods do not offer any speed advantage over full factorial data-collection approaches (such as those available in the free Google Website Optimizer tool) if you are simply planning to understand the impact of the individual variables in the test (a so-called "main effects" analysis).

If you plan on using parametric (i.e. "model building" )approaches for landing page testing you should always use full factorial data collection regardless of the subsequent analysis you plan to do. It greatly simplifies your test design, and produces better estimates of the main effects.

All parametric methods (including fractional factorial) are also outclassed by newer non-parametric testing methods such as the SiteTuners TuningEngine, which can be licensed to run your own tests in-house and have the following advantages:

  • Very large test sizes (1,000-10,000 times larger with the same data rate)
  • Much faster data collection (on the same data rate)
  • More accurate results (consider variable interactions)
  • Flexible test construction
  • No knowledge of statistics required

Hopefully this will set the record straight. If you still have an issue with this, and insist on proclaiming the superiority of fractional factorial methods, tell your statistician to call us and I will have my Chief Scientist beat them up properly.


This article originally appeared in Tim's Search Engine Watch column May 15, 2008

Recent Blog Posts:

Emotional Motivators in Landing Page Optimization

Posted by Tim Ash | Comments

People don't often make rational decisions. In fact, the capacity for abstract rational thought is only a recent evolutionary addition to our brains. We've mainly gotten by on our emotions and gut feelings. We may think we're approaching something rationally, but most of the time we use after-the-fact rationalization to...

comments | Read the full post

A Seat at the Table for Web Analytics

Posted by Tim Ash | Comments

I just got back from speaking on a multivariate testing panel at eMetrics in San Francisco.

comments | Read the full post

What Are You Converting?

Posted by Tim Ash | Comments

When most people think of "conversions," they think of large-scale or "macro" events, such as product sales or sign-ups for a service. Those are one kind of conversion, but there are many more events that fit the definition as well.

comments | Read the full post

HackerSafe & The Power of Trust

Posted by Tim Ash | Comments

I am often asked by people what sorts of items I would recommend testing on a landing page. There are no universal truths and your mileage may vary. However, there are common themes that work well across a range of industries.

comments | Read the full post

Your Baby's Ugly - Why You Need Landing Page Optimization Now

Posted by Tim Ash | Comments

All new parents think their drooling, wailing, wrinkled little midget is beautiful - and that's as it should be. The same seems to be true of landing pages. If an online marketing campaign is making money, it's taken as proof the landing pages are beautiful--and don't need further improvement.

comments | Read the full post

Interactions and the Power of Context

Posted by Tim Ash | Comments

What are variable interactions? Simply put, it is when the setting for one variable in your test positively or negatively influences the setting of another variable.

comments | Read the full post