Statistical Significance

Statistical Significance Definition

One of the most common ways researchers attempt to answer a question is by comparing two samples and seeing if there is an observed difference.

Depending on several factors, this observed difference can be either due to chance or some other significant factor. But how do we know the difference? The best way is to determine whether the observed difference is statistically significant.

Researchers are especially interested in statistical significance during hypothesis testing. Two types of hypotheses are considered in hypothesis testing: the null hypothesis (H0) and the alternate hypothesis (H1).

If an observed difference is found to be statistically significant, we can reject the null hypothesis and accept the alternate hypothesis.

Fig. 1, What are the odds, Pexels.com

Determining Statistical Significance

Determining statistical significance should first begin with finding the effect size.

Two essential things must be true about the samples taken.

• The sample must reliably represent the population, meaning there should be low variability within the group.

• The sample size must be large enough. It may be a less accurate representation of the population if it's too small.

Once the effect size is determined, we can find the value that will tell us if the effect size was just a fluke or due to some other factor. This value is called the p-value.

If this number is below the significance level or the value set at the start of the study, we can reject the null hypothesis, meaning the results we obtained were not due to chance.

Statistical Significance Formula

In order to find the statistical significance of a study, we must find the p-value. This can be complicated, so we use several different tables that do the hard part for us. However, to read these charts, there are a few things we need to understand first.

Earlier we mentioned that for the effect size to be reliable, the sample must be from a large sample and have low variability. When these two things are true it should create a curve with a normal distribution.

Fig. 2, Normal distribution curve displays a continuous probability distribution, Commons.Wikimedia.org

The next thing we need to understand for the statistical significance formula is a test statistic. Many times, researchers will find the z-test statistic. The z-test statistic essentially takes the data we collected including the sample mean, sample standard deviation, and sample value, and gives us one single value. The type of test we perform tells us which tail end of the curve we pay attention to -- lower-tailed, upper-tailed, or two-tailed test.

Fig. 3, Upper-tailed test, Commons.Wikimedia.org

Now, let's put everything together to find our p-value. Once we've found our z-test statistic, we find the point on our normal distribution curve. If it is an upper-tailed test, we are paying attention to the area to the right of the z-test statistic. The value of this area is the p-value. As we mentioned earlier, while there is a formula to find this area, it's a bit complicated. So instead, we use p-value charts or calculators to find our value.

Statistical Significance Example

Let's set up a hypothesis test as a statistical significance example. Say you want to see how many students go to college at your school compared to the national average. Here are your hypotheses:

• Null hypothesis: the observed difference between your school and the national average is due to chance.

• Alternate hypothesis: the observed difference between your school and the national average is due to something other than chance.

You set our significance level at 0.01 which means our probability that the observed difference is due to chance must be less than 0.01 before you can reject the null hypothesis. You get a z-test statistic of -2.43 and a p-value of 0.0075. This value is less than your significance level, therefore, your results are statistically significant and the null hypothesis can be rejected.