Data Collection in Multivariate Testing

There are two main design approaches used in multivariate testing: full factorial and fractional factorial. Before you get too far in planning your multivariate testing, you’ll need to understand the advantages and disadvantages of each.

Full Factorial Test Designs

Full factorial experimental designs sample data across your whole search space. If this is done properly, the subsequent analysis allows you to consider not only the main effects, but all variable interactions as well (including higher-order ones).

Because of the sheer volume of interactions tested with a full-factorial design, this type of test generally requires a lot of data, meaning that the site being tested should have a pretty high rate of traffic. But because of the exponential growth of the number of model co-efficiencies, full factorial design quickly hits its limit if you are planning to conduct an analysis of all the possible interactions. For this reason, landing page tests that use full factorial designs often have a relatively small search space.

Full factorial data collection offers a number of advantages:

• • Considers interactions. If you use full factorial data collection coupled with a complete model, you can detect all important variable interactions.

• • Unrestricted test design. You can choose any number of test variables, and arbitrary branching factors for each one.

• • Better estimation of main effects. Even if you discard the interaction data and only build a model of the main effects, your estimate of the main effects will be more accurate than with fractional factorial designs. This is because data is collected across all recipes rather than a subset of your search space.

Full factorial data is powerful, yet the analysis can be complicated and overwhelming. While many tools exist that can collect and interpret data on main effects, it’s advisable to have a strong background in statistics to determine the most meaningful variable interactions.

Fractional Factorial Test Designs

Fractional factorial multivariate testing allows you to simultaneously test several key elements of your website or landing pages. In effect, it’s like running several A/B split tests at once, but with total traffic requirements that are significantly less than would be necessary for an equivalent number of separate A/B split tests. Fractional factorial test designs have the advantage of compressing the amount of data (website traffic) required, while still giving you the benefits of multiple simultaneous A/B split tests.

Fractional factorial designs fall under the design of experiments (DOE) umbrella. DOE is a systematic approach to getting the maximum amount of useful information about the process that you are studying, while minimizing the amount of effort and data collection required. Commonly known by the name Taguchi Method, fractional factorial testing requires you to determine upfront what elements and variations you will test. “Fractional” means just that: you will be collecting data on only some, or a fraction, of the possible recipes, or variables.

One feature of fractional factorial testing is that it allows you to explicitly define the interactions among the variables that you want to study and examine. But this can be a disadvantage as well, because you’ll need to make guesses about which interactions will be important and then build those assumptions into your model upfront.

There are other key disadvantages to fractional factorial test designs:

• • The Frankenstein effect. Because you are collecting data about several variables simultaneously yet independently, you run the risk of creating a final page recipe comprised of all your winning variables, yet find that combined these variables don’t perform well. It is difficult to determine the best results without considering how the variables interact with one another.

• • Restrictive test design. Your test design must follow a certain pattern, in terms of the number of variables and their branching factors. As a result, you are forced to either stick with well-known “standard” designs from statistical textbooks, or construct your own (with the help of statisticians).

• • Throttling is difficult. If you throttle your data collection rates and don’t devote equal bandwidth to each recipe in your test, your analysis will be invalid.

However, by limiting data collection to fewer elements, a fractional factorial test can be completed more quickly and with lower traffic rates than a full factorial test. For those desiring a more iterative approach to their testing, fractional factorial testing may provide some quick answers on key page elements.

Summary

As with most things in life, there is an inherent trade-off among various multivariate test constructions. Full factorial parametric designs do not scale very well, but get more complete information about the exact relationship among all main and interaction effects tested. Fractional factorial designs can scale to larger search spaces, but make assumptions about the underlying process that may not be valid and may actually lead you astray.

This article originally appeared in Tim’s ClickZ column June 6, 2011