Multivariate Test Construction

There are two primary types of multivariate test designs: unconstrained designs and constrained designs. Let’s discuss these further.

Unconstrained Designs

Unconstrained design test variables can be created and displayed independently of each other on your landing page. For instance, if all variables can be displayed in separate locations on your landing page, they are usually unconstrained. Most basic landing page tests involve unconstrained designs.

This does not mean that there are no variable interactions among variables in unconstrained designs. Some variables may interact with each other. For example, the call-to-action text on the button may interact with the button color. Imagine that the button text says “go.” On a green button this makes sense.

However, the same text displayed on the red button would be confusing since people commonly associate red with the word “stop.” So you should look for possible interactions among all the possible variable value combinations of V3 and V4, but your choices of the exact wording of the alternative calls-to-action text can probably be considered independently of your choices about alternative button colors.

Constrained Designs

Constrained designs involve conditional rules for constructing certain recipes. In other words, some of the allowed values for a particular variable are contingent on the setting of others, or can exist only under certain conditions. Under such circumstances, take special care to properly define and structure your variables. You may also have to make sure that you sample appropriately during your test and do not accidentally create improper recipes for presentation to your visitors.

Let’s consider a simple example. Assume that you have a two-page registration form. You want to test an alternative design for the second page, but you also want to consider a design where all of the form fields are moved onto the first page (i.e., there’s no need for a second page at all).

One possible test construction approach is as follows:

V1 – first page contents

a – original first page

b – extended first page (containing all original first and second page fields)

V2 – second page contents

a – original second page

b – alternative second page design

c – no second page (all content moved to V1b first page)

Under this unspecified constraint approach, you will need to keep track of the fact that V1b and V1c can only appear together, and enforce it through rules that are external to the variable definition itself.

Another common solution is to flatten the constrained variables and create a single variable containing values for each allowable design.

V1 – registration process

a – original first and second pages

b – original first page and alternative second page

c – extended one page (containing all original first and second page fields)

As a real-life example of a constrained design, ScanAlert, the creators of the Hacker Safe website trustmark, worked with my company to conduct multivariate testing of its trustmark to qualifying prospects and customers of Hacker Safe. The goal was to find the biggest conversion rate improvement possible by adding the trustmark. The company tested six different logos in multiple positions on the client’s landing page.

We tested against the original client landing page, which did not contain a trustmark. Obviously, if the trustmark is not present on the landing page, the specific logo chosen and its position becomes meaningless. To run these tests, we chose the specified constraint test construction approach. In other words, we reserved one variable specifically for the constrained condition, and a placeholder variable value for it in all other appropriate variables:

V1 – trustmark presence (branching factor = 2)

a – no trustmark

b – trustmark

V2 – trustmark logo (branching factor = 7)

a – no logo (placeholder for constraint in V1)

b – red-horizontal logo

c – red-vertical logo

d – black-horizontal logo

e – black-vertical logo

f – white-horizontal logo

g – white-vertical logo

V3 – trustmark location (branching factor = 5)

a – no location (placeholder for constraint in V1)

b – location no. 1

c – location no. 2

d – location no. 3

e – location no. 4

Based on the branching factors here, you might assume that this test has 70 distinct recipes (2 – 7 – 5). However, it only has 25 (one original plus 24 trustmark logo and position combinations). Besides the baseline recipe (specified by aaa), no other recipes are allowed to contain any a’s.

The Bottom Line

Unconstrained test designs are far easier to implement. You can set a different number of values for each variable so the branching factor is determined simply by the number of good options you want to test for each variable. Unconstrained designs go hand-in-hand with full factorial data collection.

Constrained test designs require you to predefine the number of variables and their branching factors. These designs are used with fractional factorial data collection and require you to choose from a standardized test construction.

Either way, it’s important to choose wisely, as your test design will limit your options for data collection (full factorial vs. fractional factorial) as well as data analysis (parametric vs. non-parametric).

This article originally appeared in Tim’s ClickZ column June 20, 2011