1

According to Wikipedia,

A result has statistical significance when it is very unlikely to have occurred given the null hypothesis

does that mean if our alternative hypothesis is true we will call our results "Statistically Significant"? which certainly shouldn't be the case.

Alexis
  • 29,850
christopher
  • 450
  • 2
    alternative hypothesis is not needed for the concept of significance – carlo Apr 12 '20 at 16:44
  • 1
    @carlo: Isn't statistically significant always in reference to some hypothesis test? Can you explain. – ColorStatistics Apr 12 '20 at 16:57
  • 1
    yes, statistical significance is a concept related to Fisher's theory of hypothesis testing. That theory requires a null hypothesis to undergo the test, and no second alternative hypothesis needs to be considered. – carlo Apr 12 '20 at 17:12
  • 2
    I am not sure I am following. Doesn't having a null hypothesis automatically imply you have an alternative hypothesis as the negation of the null? What is the point of testing the null hypothesis then? What happens if you reject the null? Seems to me you'd have a silent alternative hypothesis then. Doesn't seem to be a useful approach to hypothesis testing. – ColorStatistics Apr 12 '20 at 17:36
  • 1
    Without an alternative hypothesis, how do you distinguish between 1-sided and 2-sided hypothesis tests? – ColorStatistics Apr 12 '20 at 17:43
  • well, that is a way of putting it, but this creates confusion with Pearson's hypothesis testing theory, and with its modern days inheritance, like the concept of test power. those require a well defined alternative hypothesis which is not simply the negation of the null. – carlo Apr 12 '20 at 17:44
  • 1-sided or 2-sided tests have a different null hypothesis, it's not simply a matter of alternative hypothesis – carlo Apr 12 '20 at 17:45
  • With a null hypothesis of mu=0, you can have 3 different alternative hypotheses: 1) mu>0, 2) mu<0, and 3) mu not equal to 0. They are all distinguished by the alternative hypothesis. – ColorStatistics Apr 12 '20 at 17:48
  • for a one sided test your null hypothesis is typically on the form of $\mu \le 0$. – carlo Apr 12 '20 at 18:13

2 Answers2

8

does that mean if our alternative hypothesis is true we will call our results "Statistically Significant"?

No. If the alternative really is the case, we can still fail to reject the null resulting in a type II error (or a false negative).

The mathematial definition of significance is straight forward. When our p value is less than our type one error rate (typically p<0.05), then we call our result statistically significant. Translating this into an inference about the real world is where the trouble usually arises.

I like to think of hypothesis tests as a dilemma. You start with an initial assumption about the world (e.g. that the null truly is the case and that your assumptions about the data generating processes really are true). You perform your test and get a p-value. The interpretation of that p value is similar to what you have in bold; it is the probability that you see a result at least as extreme if not more extreme given that the null is true and your modelling assumptions are true. Now for the dilemma. Assuming the p-value is smaller than small enough (assuming you have some way of choosing what small enough means) you have just observed something quite improbable under the null. So you have two choices:

  • Conclude that you have not observed anything which would falsify your initial beliefs about the world and accept that you have seen something incredibly rare.

  • Conclude that one of your beliefs about the world must have been wrong because you have seen something incredibly rare assuming your initial beliefs were true.

Often times, we opt for the second and hence we reject the null. In my opinion, that is what statistical significance means. In a phrase,

"statistical significance is the observation of a test statistic which is sufficiently improbable under the null hypothesis, putting us in the dilemma described above in which we opt to conclude our initial beliefs about the world were in fact incorrect".

This isn't a perfect definition and I'm open to changing that should anyone care to improve it.

Demetri Pananos
  • 36,121
  • If p value is less than some predefined level how can we conclude that our initial beliefs are correct? Shouldnt we immediatly reject null or you are refering to the fact that it is possible to get a result by chance? – christopher Apr 12 '20 at 17:17
  • The choice in the dilemma is really up to you. If you get a p value less than 0.05 you can still choose not to reject the null. We have all just agreed that when p is less than 0.05, then we choose to conclude our initial beliefs were wrong. – Demetri Pananos Apr 12 '20 at 17:27
  • So the result means nothing ? We have to define what we want and statistical significance could mean different for everybody? Some would would chose to reject and some wouldnt. – christopher Apr 12 '20 at 17:34
  • That is precisely the case! We are free to define alpha which in turn determines what is "statistically significant". The 0.05 threshold just happens to be something that a lot of people use. – Demetri Pananos Apr 12 '20 at 18:10
0

More or less, yes.

You call the descriptive quantity of interest *statistically significant" if the associated null hypothesis is rejected at the prespecified level.

Consider e.g. a randomized study comparing a new drug versus a placebo with null hypothesis $$ H_o: \mu_\Delta = 0, $$ where $\mu_\Delta$ is the true mean difference between the two treatment options. The hypothesis should be tested at level 0.05.

After the study has been conducted, you estimate $\mu_\Delta$ by the empirical mean difference, 0.3, say and calculate a p value (0.02 say) corresponding to the stated null hypothesis. Then you would call your empirical effect 0.3 as statistical significant at the level of 0.05.

Since usually, the alternative hypothesis is the contrary of the null hypothesis, my answer is "more or less yes".

Michael M
  • 11,815
  • 5
  • 33
  • 50