Multivariate testing is the testing of several page elements simultaneously, with the aim of finding out not only which variant is better, but also how the elements affect one another. The difference from ordinary two-version comparisons lies in the number of combinations: with three elements each having two variants, eight versions are created, among which traffic is split. This is precisely the main limitation – the required sample grows with the number of combinations, and most websites do not have enough visits for this, so the test never reaches statistical significance and is evaluated incorrectly. In practice, this means that with ordinary traffic it is better to test one significant change at a time, and to leave multivariate testing to websites with high volume. Before launching, always calculate the required sample size.
See also: Statistical significance of a test, Creative fatigue, Micro-conversion.
