0

Post Edit: (I believe the question I ask here has not been answered in other questions regarding p-value on stats.stackexchange, mine in synthesis asks "does a low P-value mean this (read question)?" rather than when to accept or reject null hypothesis using p-value and its relation to t stat which I fully understand, this is why I have not accepted duplicate suggestions)

I'm studying Econometrics (undergraduate level), and I can't grasp what a low p-value suggests, let me explain:

supposing we arbitrarily decide that our rejection region for a two tailed hypothesis test of a normal distribution $N(0,1)$, is set at $|{t}|>1.96$, we affirm that any t stat. greater than $1.96$ would result into a rejection of our $H_{0}$ in favor of the alternative Hypothesis.

P-value being the probability, under $H_{0}$, of getting a $|t|>|\hat{t}|$, $\hat{t}$ = the t statistic estimated form our sample, we would reject $H_{0}$ in favor of the alternative Hypothesis in case P-value was any lower than p-value =$0.05$; (so any $|\hat{t}|>1.96$ would mean rejection of $H_{0}$ because $Pr=[|{t}|>1.96]$ would be < than .05).

Question: So, does a low p-value (in our example lower than .05) in practical terms, tells us that we obtained a t stat. "so rare", that is very hard to sustain this happened by chance and is due to the data used in our sample? And consequently we now believe the cause of a low P-value is in the null Hypothesis not being true, rather than having used a rare sample, correct?

Thanks to anyone helping.

Adriano Pollio
  • 15
  • 1
    @mkt, sooner or later, we can conclude that very question can be taken as canon and all other p values questions can be closed as duplicate, imo. – User1865345 Aug 26 '22 at 15:22
  • 2
    "how likely we are to have other samples dealing a t stat. lower than the one that our sample has dealt" is incorrect unless "how likely" is understood as meaning assuming the null hypothesis. But I agree with the previous suggestions that these issues are well covered in the referenced threads on p-values. – whuber Aug 26 '22 at 15:35
  • 2
    I voted to close, not because of duplicate reasons (which it might be) but because I can not make much sense of the sentence in the question (which is more than 50 words long). – Sextus Empiricus Aug 26 '22 at 17:57
  • 1
    A lot of confusion about p-values arises because it is a versatile tool while often it is not regarded like that and considered from a single perspective. Then your question sounds to me like "When people use a hammer to hit a nail into a piece of wood, are they hitting the hammer because the want to kill the nail?" And you make it more confusing but adding a lot together "When people use a hammer to hit a nail into a piece of wood, are they hitting the hammer because the want to kill the nail, and they don't have a gun, and they prefer to hold it in their left hand?". – Sextus Empiricus Aug 26 '22 at 18:11
  • 1
    I'm afraid I still don't understand your question. Your title asks what it 'suggests'. Your update edit says that your question is 'does a low P-value mean X'. Your actual question text is basically 'how do I interpret a p-value?'. All of these are addressed by the duplicates. If your question is really different, it's not coming through in the text. – mkt Aug 26 '22 at 19:17

1 Answers1

4

When we do a hypothesis test, we begin with a supposition about how the world is for the purposes of logical deduction; that the means are the same, that the ratio of the variances is 1, and so on. We call this "the null hypothesis". We usually contrast this against some other possible state of the world (e.g. the means are difference) which might be the case. We call this "the alternative hypothesis".

Then, we go about with our experiment and we calculate the probability of seeing a result at least as extreme as what we saw, given our supposition about the world is true, and all our other underlying assumptions are true. This is called "the p value".

Now suppose we see a small p value. Then we are placed in the following dichotomy. Either:

  • The null hypothesis (our supposition about the world) is true and we've just seen something incredibly rare, or

  • Our initial supposition about the world was false.

Edit: Note that I have been careful in my description of the definition of the p value, and that it is possible that, for example, the extreme probability comes from a violation of one of the assumptions rather than the null being false. Indeed, the reason behind the improbable result is underdetermined. Implicit in this description is that we do our due diligence in verifying these assumptions are defensible vis a vis plots and or substantive knowledge about the problem.

Put simply, I am assuming all assumptions are correct and that we have not made a calculation error. Those are possible, but muddy the story in my opinion.

We often just choose to believe we were wrong about the null hypothesis when this happens. This is called "rejecting the null".

So what a small p value suggests is that the probability that we would see a result at least as extreme as the one we saw assuming the null is true is very small. It could be the case we were right and that just got extremely lucky, but we often choose not to believe that.

Demetri Pananos
  • 36,121
  • It might help to clarify your second sentence, because it is overwhelmingly common for the belief we begin with to be framed as the alternative hypothesis and not the null. – mkt Aug 26 '22 at 15:17
  • THANK YOU so much, this is the perfect answer to my question and you understood perfectly what my doubts were. I was so in disbelief that I could not find an answer anywhere as clear as yours, I tend to get really stressed out when I don't have perfect clarity over a topic. I hope your answer helps out anyone with the same problem. Thanks again! – Adriano Pollio Aug 26 '22 at 15:18
  • @mkt I've edited for clarity – Demetri Pananos Aug 26 '22 at 15:20
  • @AdrianoPollio, if Demetri's post answered your query, please accept the same. – User1865345 Aug 26 '22 at 15:21
  • Hi @DemetriPananos. + 1 from me. However, could you insert a little about the frequentist assumptions here? I believe this would give a more complete perspective to the OP on the theoretical basis of the p-value. – EB3112 Aug 26 '22 at 15:26
  • @EB3112 Which assumptions did you have in mind? – Demetri Pananos Aug 26 '22 at 15:32
  • 3
    (1) Re "we begin with a belief": Certainly not! In most cases $H_0$ is contrary to our belief. There's a difference between a supposition entertained for the purpose of logical deduction and a "belief." (2) Characterizing the situation as a dichotomy is a gross oversimplification, because many other plausible explanations exist for why a p-value might be small. For instance, possibly neither of the null or alternate hypothesis is true. Or maybe the model is just plan incorrect. Or we made a computational error. Etc. Some of these possibilities are sometimes termed "Type III errors." – whuber Aug 26 '22 at 15:39
  • 2
    @whuber Fine, I admit it to be a supposition rather than a belief. Granted as well that the rejection of the null may come from violated assumptions, but the statistical workflow encourages people to check those assumptions vis a vis something like a qqplot or histogram of residuals, etc. I can edit the answer to explicitly mention these. – Demetri Pananos Aug 26 '22 at 15:48