Statistically significant changes

When analyzing your data, it can be easy to use words such as "significant difference", but in statistics, "significant difference" means more than just a big change from the previous period. In this article, we'll go through what statistical significance is, and when to test the significance.

What is statistical significance?

For any statistical data, there is a margin of error. In other words, your data will never be exact (unless you can test the entire population) but an approximation. The margin of error is the range in which the results should fall if you repeat the same study several times. The larger the sample size (how many participants), the smaller the margin - which means that your results will be more precise. 

The purpose of a significance test is to see if the change between samples (e.g. two segment or two different time periods) falls within the margin of error. If it does, it means that the change you see is part of the normal variation. If it does not, it means that there is a difference between the samples. This difference could either be caused by changes to the population (e.g. the website is attracting different visitor groups) or by changes to the study (e.g. reduced trigger time means you are capturing a wider segment of the visitors). Note that the significance test does not tell how big the difference is.

When should you do a significance test?

For a normal web survey with the purpose of understanding the user experience, you don't need to do any significance tests. By focusing on the significance, there is also a risk of shifting the focus away from the key results. The opinions and experiences expressed in the answers are more important than the accuracy of the data. This doesn't mean that significance tests are useless or harmful to your analysis, but they are not essential. 

A web survey usually also has a pretty good response rate, especially compared to an election poll, and the margin of error is therefore smaller. We at Extellio have worked with web surveys for many years and the results tend to be quite stable from year to year for our customers, unless the customers have made changes to their website, to the survey, or are attracting new visitors groups (e.g. through campaigns or by acquiring a different company).

For studies where the sample size is a very small portion of the population, such as election polls, or the accuracy of the data is of higher importance, such as evaluating the effects of a certain measure, significance tests become helpful tools in the analysis.