In at least certain strands of "classical statistics", there seems to be a shared commitment to the following claim:
For any null hypothesis H, if the p-value of H and the observed test statistic is below the significance level alpha, then we should reject H.
For example, this is an excerpt from the Wikipedia page on statistical significance:
Statistical significance plays a pivotal role in statistical hypothesis testing. It is used to determine whether the null hypothesis should be rejected or retained. The null hypothesis is the default assumption that nothing happened or changed.[36] For the null hypothesis to be rejected, an observed result has to be statistically significant, i.e. the observed p-value is less than the pre-specified significance level alpha
Some clarification questions:
- Rejected by whom? The entire scientific community?
- What is the meaning of this "should"? Is it something like "we would reject H, if our goal was to promote the progress of science as well as possible"?
- What does it mean to "reject" H? Should I interpret this epistemically, as in "our evidence against H is sufficiently strong that we can pretend or act as though H is certainly false"?
Most importantly for me:
- What justifies this normative principle?
