Arbitrary and non-arbitrary levels of significance
The arbitrary significance levels are decided before calculating the contrast statistic and the non-arbitrary significance levels depend on the value that the contrast statistic takes, both depend on the distribution that the data follow.
In other words, the arbitrary significance levels will always be the same for different values of the test statistic and the non-arbitrary significance levels will be different for different values of the test statistic.
When a concept is pointed out, the characteristic of being arbitrary means that the value of that concept is chosen by the researcher. a priori (before) doing the experiment without relying on any related information.
P-value and elephants
For example, suppose we want to test the number of elephants in a meadow.
P-value and elephants
Before we see the meadow and the elephants that actually exist, we suppose a priori the number of elephants. We say that there can be 10 elephants. So, we go to the meadow and count the number of elephants that we see: 1, 2, 3, 4, 5, 6 and 7.
Our null hypothesis was that the number of elephants in the meadow was equal to 10 and our alternative hypothesis was that there were less than 10. So, given the elephants there are, we would reject the null hypothesis. But… What if there are 3 more elephants in the meadow but they are hiding behind the trees? We would be rejecting our null hypothesis when it could be true if, instead of counting the elephants, we had calculated the maximum number of elephants that the meadow can accommodate.
The 10 elephants chosen at the beginning has been totally arbitrary because we have not seen the size of the meadow and, therefore, we do not know if 10 elephants is a lot or a little.
On the other hand, if, given the size of the meadow, we calculate the maximum number of elephants that it can accommodate, we will know what the maximum value is so as not to reject the null hypothesis. So finding the real number will be much easier.
The same is true for the 1%, 5%, and 10% significance levels compared to the p-value. In many contrasts we choose the level of significance without taking into account any information other than the distribution. Normally 5% is used as the level of significance (alpha), leaving 95% of the sample within the confidence interval.
The problem of assigning the significance level arbitrarily is the same problem we have with the elephant example. If we believe that it is correct to apply 5% (significance level), we may reject the null hypothesis when the minimum to be rejected is 2% (p-value). We would incur erroneous results simply by setting 5% instead of the minimum value to be rejected (2%).
In other words, we are concluding that there are less than 10 elephants in the grassland but in reality there are 3 more elephants but they are hidden. So, it is much faster to calculate what is the maximum or minimum significance level for which we would not reject or we would reject the null hypothesis.
If value - p <significance level => Rejection H0.
If value - p> level of significance => No rejection of H0.