Multivariate Testing

The purpose of multivariate testing is to simultaneously gather information about multiple variables, and then conduct an analysis of the data to determine which recipe results in the best performance.

Multivariate testing approaches differ on two important dimensions:

  • How the data is collected
  • How the data is analyzed

Data Collection - Factorial vs Fractional Factorial

Multivariate test data can be collected in a full factorial or fractional factorial fashion

  • Full factorial experimental designs sample data across your whole search space. The subsequent analysis allows you to consider not only the main effects, but all variable interactions as well (including higher-order ones). Full factorial parametric designs do not scale very well, but they get more complete information about the exact relationship among all main and interaction effects tested.
  • Fractional factorial designs (also called “DOE” - design of experiments) is a systematic approach to getting the maximum amount of useful information, while minimizing the amount of effort and data collection required. This is accomplished by making simplifying assumptions about the possible form of the parametric model for subsequent analysis. Fractional factorial designs can scale to larger search spaces, but make assumptions about the underlying process that may not be valid and may actually lead you astray. Fractional factorial data collection and test design precludes non-parametric analysis.

Data Analysis - Parametric vs Non-Parametric

  • Parametric data analysis in landing page optimization builds a model of how the variables tested (the “independent variables”) impact the conversion rate (the “dependent variable”). For each recipe in your search space, the model will produce a prediction of the expected conversion rate (or other optimization criterion of interest). There are different forms of parametric analysis, too: some take complex variable interactions into account, while others do not. But whatever form you choose, it's critical to run follow-up A/B split tests between the predicted best challenger recipe and the original baseline recipe. Why? Because unless you happened to have sampled data on the exact recipe predicted by the model to be the best, you don't really know if the prediction will hold up.
  • Non-parametric data analysis, by contrast, does not try to build a model based on the input variables. Instead, non-parametric methods try to identify the best challenger recipe, but without being able to tell anything about why it is the best, or exactly how much better it is than your baseline.

The two approaches are unrelated and are answering different questions. They are both a recognition of the fundamental reality that only so much useful information can be extracted from your data collection sample. The only question is what you want to do with the data. You can try to create a general model of the output variable and describe it in terms of the input variables, or you can find the best individual recipe and not know why it is the best.

Constructing a Multivariate Test

There are two primary types of multivariate test designs: unconstrained designs and constrained designs.

  • In an unconstrained design, test variables can be created and displayed independently of each other on your landing page. For instance, if all variables can be displayed in separate locations on your landing page, they are usually unconstrained. Most basic landing page tests involve unconstrained designs.
  • Constrained designs involve conditional rules for constructing certain recipes. In other words, some of the allowed values for a particular variable are contingent on the setting of others, or can exist only under certain conditions. Under such circumstances, take special care to properly define and structure your variables. You may also have to make sure that you sample appropriately during your test and do not accidentally create improper recipes for presentation to your visitors.