4

I understand that the p-value is defined as the probability to obtain a "more extreme" value of $w$ if $H_0$ is true, i.e.

$p=P(|W| > |w| \ |H_0 $is true$)$

and the "significance level" $\alpha$ is a threshold to decide if it is "admissible" that $w$ comes in null hypothesis setting, i.e.

$p < \alpha \Rightarrow$ reject $H_0$.

However, It is hard for me to understand the meaning of hypothesis test when $\alpha$ is interpreted as a probability, specifically:

$\alpha=P($reject $H_0 | H_0 $is true$)$.

In this case, I will have:

$P(|W| > |w| \ |H_0 $ is true$) \ < P($reject $H_0 | H_0 $is true$)\Rightarrow$ reject $H_0$

Therefore I'm saying that, if $H_0$ is true, the probability of $w$ is less than the probability of rejecting $H_0$, therefore reject $H_0$. Since, for example, I obtained $w$, its probability is lower than the probability of in rejecting $H_0$, so reject. In other words, it is more probable that I reject when I shouldn't, respect to the fact that $w$ has been sampled, so reject. This seems to me that it is more probable that I'm making a reject when I shouldn't, or there is something that escapes me. Furthermore, I'm comparing two different distributions, i.e. $P(W)$ and $P(\text{ reject } H_0)$, that seems to me two different objects, therefore I don't understand the point of comparing them.

This question can be viewed similar to The rationale behind the "fail to reject the null" jargon in hypothesis testing?, but in my opinion is different (it regards also the type II error, which is not the subject of my question) and also to What is the meaning of p values and t values in statistical tests? (which see $\alpha$ simply as a threshold, and not as a probability): Maybe a similiar question is intuition/logic behind comparing p-value and significance level but the given answer is not fully satisfactory for me, and the question has been closed since considered too similar to the other questions.

So, what is the logical meaning in rejecting the null-hypothesis when $\alpha$ is viewed as the probability to make a Type I mistake?

User1865345
  • 8,202
volperossa
  • 747

1 Answers1

7

Until you study Bayesian inference this will not make as much sense to you. But you started off incorrectly. With continuous data the probability of achieving any one value of a test statistics (or of a summary measure the test statistic is based on) is zero. So the p-value is the probability of getting a test statistic that is more extreme than the observed one if $H_0$ is true and the model is correct. To read more about the enormous difference between $\alpha$ and decision errors, read this.

$\alpha$ isn't the probability of making a mistake. This is one of the most common interpretation errors and is at the root of a lot of unclear thinking by a lot of practitioners. The term "type I error rate" is a misnomer that started us off on the wrong foot a century ago. It's not a rate and is not an error probability. That's why I've moved to the term "type I assertion probability $\alpha$ in its stead. It's just an assertion trigger probability. No conditional probability that assumes $H_0$ is true can inform you about the veracity of $H_0$.

With Bayesian posterior probabilities you compute probabilities of any assertion you have and use direct reasoning. If you decide to act as if an effect is positive when P(effect > 0 | data, prior) = 0.98 you automatically get a decision error probability to carry along, which is simply 0.02.

Frank Harrell
  • 91,879
  • 6
  • 178
  • 397
  • 2
    I'm not a Bayesian and have only studied it a little, but your answer makes good sense to me. Maybe the little that I've studied it is enough. – Peter Flom Aug 16 '23 at 11:34
  • with "more extreme" you mean "minor than" ?i.e. $P(W < w |H_0 \text{ true })$? – volperossa Aug 16 '23 at 12:04
  • 1
    It's generally stated as $\Pr(W > w)$ but you can do it either way if you want a one-sided test. But most tests are two-sided so use something like $\Pr(|W| > |w|)$. – Frank Harrell Aug 16 '23 at 12:21
  • 1
    thank you. I tried to correct my question accordly, but still there is something missing in all the reasoning. More precisely, I don't understand what you mean with "assertion probability" – volperossa Aug 16 '23 at 12:56
  • 2
    In simplest terms one triggers an assertion of an effect (rejection of the supposition of no effect) when for example p < 0.05. The probability of making such an assertion under $H_0$ is 0.05 if p-values are accurately computed and the data model holds. – Frank Harrell Aug 16 '23 at 14:00
  • therefore you see alpha not as a probability ma simply as a threshold, right? – volperossa Aug 16 '23 at 21:22
  • 3
    I have a semantic issue here. If in mathematical symbols we write $$\alpha=P({\rm reject }, H_0 | H_0, {\rm is, true})$$, then is it not grammatically correct to translate this in words as "the probability of rejecting $H_0$ given that $H_0$ is true", which is an example of "making a mistake"? – Alecos Papadopoulos Aug 17 '23 at 01:25
  • It's not an example of 'making a mistake' for a subtle reason. When people say "what is the chance I was wrong" they are implicitly using an unconditional probability, i.e., not assuming truth or non-truth of $H_0$. Pr(mistake) = unconditional probability of being wrong about an effect, not a probability of being wrong when by definition any assertion of a positive effect is wrong. – Frank Harrell Aug 17 '23 at 15:06
  • 1
    P.S. If you were willing to assume $H_0$ when making a judgement about a decision you don't need any data at all to know about any mistakes. – Frank Harrell Aug 17 '23 at 15:33