Segmented test evaluation means assessing the result not only overall but also by groups of visitors – by device, traffic source, visitor novelty, or country. The reason is that the overall result can hide two opposite effects: a change that helped significantly on desktop may hurt on mobile, and taken together it looks like a null result. Segmentation, however, needs to be approached carefully. The more groups examined, the greater the probability of finding a random difference with no real significance, and smaller segments also have a smaller sample. That's why segments are defined in advance together with the hypothesis, not added afterwards while searching the data for a favourable result. A finding in a single segment is therefore treated as a prompt for a further test, not as a confirmed result.
See also: Statistical significance of a test, Website A/B Testing, Mobile conversion rate.
