Email A/B Testing compares two variants of a mailing that differ in one element – most often the subject line, sender, main image or call to action. Part of the database receives one version, part the other, and the winner is either sent to the rest or used as a learning for the future. A meaningful test requires a sufficient sample size: with a database of a few hundred contacts, the difference in open rate is usually random, and deciding based on it leads to mistaken conclusions. Only one variable should be tested at a time, and the result should be judged against the campaign's goal, not the open rate – a higher open rate with fewer orders is not a win. It is worth recording findings, otherwise the same tests get repeated indefinitely.
See also: Statistical significance of a test, Email subject line and preheader, Multivariate testing.
