Statistical significance tells you how likely it is that the observed difference between two variants arose by chance. Without it, test results are misleading: with a small number of visits, the variants keep swapping first place from day to day, and whichever moment is declared the end of the test will give a different answer. The most common mistake is precisely ending the test early, at a moment when the favoured variant is ahead. Before launching, therefore calculate the required sample size from the current conversion rate and the difference you want to be able to detect – with low traffic, it often turns out that testing that particular element does not make sense and it is better to test a more substantial change. Let the test run for full weeks so that it covers every day, and do not change other things on the page while it is running.
See also: Geo experiment, Incrementality, Conversion window.
