What sense does it make to compare p-values to each other?

Question

I have two populations (men and women), each containing $1000$ samples. For each sample I have two properties A & B (first year grade point average, and SAT score). I have used a t-test separately for A & B: both found significant differences between the two groups; A with $p=0.008$ and B with $p=0.002$.

Is it okay to claim that the property B is better discriminated (more significant) then the property A? Or is it that a t-test is just a yes or no (significant or not significant) measure?

Update: according to the comments here and to what I have read on wikipedia, I think that the answer should be: drop the meaningless p-value and report your effect size. Any thoughts?

No problem: if you feel that the (minor) edits I made changed your question in any meaningful way, please feel free to correct them. — whuber, Jan 20 '12 at 21:13
What's the outcome you measured? (i.e. what is it that differs, between the groups defined by A/not A, or B/not B?) Is it measured on all 1000 samples, or are some missing? — guest, Jan 20 '12 at 21:37
Reporting the two different effect sizes, or confidence intervals for the two different effect sizes, would be a good idea. It would be easier to interpret this if the outcome in each of your two datasets was the same (is it?). — Peter Ellis, Jan 21 '12 at 20:21
You can show statistical significance and effect size very conveniently by use of a forest plot! Presenting 95% CIs means that you're using 4 numbers instead of 2, but as everyone is alluding to, it sufficiently represents the extent of information necessary to compare experiments. — AdamO, Jun 12 '14 at 18:49

amoeba · Accepted Answer · 2016-12-01T17:59:18.067

Many people would argue that a $p$-value can either be significant ($p< \alpha$) or not, and so it does not (ever) make sense to compare two $p$-values between each other. This is wrong; in some cases it does.

In your particular case there is absolutely no doubt that you can directly compare the $p$-values. If the sample size is fixed ($n=1000$), then $p$-values are monotonically related to $t$-values, which are in turn monotonically related to the effect size as measured by Cohen's $d$. Specifically, $d=2t/\sqrt{n}$. This means that your $p$-values are in one-to-one correspondence with the effect size, and so you can be sure that if the $p$-value for property A is larger than for property B, then the effect size for A is smaller than for property B.

I believe this answers your question.

Several additional points:

This is only true given that the sample size $n$ is fixed. If you get $p=0.008$ for property A in one experiment with one sample size, and $p=0.002$ for property B in another experiment with another sample size, it is more difficult to compare them.
- If the question is specifically whether A or B are better "discriminated" in the population (i.e.: how well can you predict gender by looking at the A or B values?), then you should be looking at effect size. In the simple cases, knowing $p$ and $n$ is enough to compute the effect size.
- If the question is more vague: what experiment provides more "evidence" against the null? (this can be meaningful if e.g. A=B) -- then the issue becomes complicated and contentious, but I would say that the $p$-value by definition is a scalar summary of the evidence against the null, so the lower the $p$-value, the stronger the evidence, even if the sample sizes are different.
Saying that the effect size for B is larger than for A, does not mean that it is significantly larger. You need some direct comparison between A and B to make such a claim.
It's always a good idea to report (and interpret) effect sizes and confidence intervals in addition to $p$-values.

Good points about monotonicity and good final 3 points. Now, re: the "you can be sure" statement: true enough for the sample, but "significantly so"? (I.e., with trustworthy implications for the population?) You did address this briefly in #2. A fuller treatment of this would be welcome. Cheers ~ — rolando2, Feb 18 '15 at 00:50
This is right, but I also tried to make clear that it is only necessarily right in this case (you also note this). I think Michelle was making a worthwhile point that you should not in general use p-values in this way. — gung - Reinstate Monica, Feb 18 '15 at 01:06
(-1) The body of this post is correct, but the opening sentence ("Many people would argue that...it does not make sense to compare two $p$-values between each other. This is wrong.") is too easily misconstrued as generic advice, when in fact it only holds in special cases, such as the one here. — Andrew M, Dec 01 '16 at 17:51
@AndrewM Perhaps. I have edited the beginning of my answer. See if you like it better now. — amoeba, Dec 01 '16 at 17:59

score 0 · Answer 2 · answered May 29 '18 at 02:18

Thanks to whoever just downvoted me, as I now have a completely different answer to this question.I have accordingly deleted my original answer as it is incorrect from this perspective.

In the context of this question, which is only dealing with the question "was A or B a better discriminator in my study", we are dealing with a census and not a sample. Thus, the use of inferential statistics such as those used to produce p-values are irrelevant. Inferential statistics are used to infer population estimates from those we obtain from our sample. If we do not wish to generalise to a population, then those methods are unnecessary. (There are some specific issues around missing values in a census, but those are irrelevant in this situation.)

There is no probability of obtaining a result in a population. We obtained the result we got. Therefore, the probability of our results is 100%. There is no need to construct a confidence interval - the point estimate for the sample is exact. We're simply not having to estimate anything at all.

In the specific case of "which variable worked better with the data I have", all one needs to do is look at the results in simple summary form. A table may be sufficient, maybe a graph like a box plot.

score -1 · Answer 3 · answered Jun 12 '14 at 18:35

Added an answer as it was too long for a comment!

Michelle has a good response, but the many comments show some common discussions that come up about p-values. The basic ideas are the following:

1) A smaller p-value doesn't mean a result is more or less significant. It just means that the chances of getting a result at least as extreme are less likely. Significance is a binary outcome based on your chosen significance level (which you choose before you run the test).

2) Effect size (often standardized to #'s of standard deviations) is a good way to quantify "how different" two numbers are. So if Quantity A has an effect size of .8 standard deviations and Quantity B has an effect size of .5 standard deviations, you'd say that there's a larger difference between the two groups in Quantity A than in Quantity B. Standard measurements are:

.2 standard deviations = "small" effect

.5 standard deviations = "medium" effect

.8 standard deviations = "large" effect

But given fixed sample size, p-value is directly monotonically related to the effect size! — amoeba, Feb 17 '15 at 15:41

score -1 · Answer 4 · answered Jan 31 '12 at 01:07

-1

You get a difference in p, but it is unclear what that difference means (is it large, small, significant?)

Maybe use bootstrapping:

select (with replacement) from your data, redo your tests, compute difference of p's (p_a - p_b) , repeat 100-200 times

check what fraction of your delta p's is < 0 (meaning p of A is below p of B)

Note: I have seen this done, but am not an expert.

answered Jan 31 '12 at 01:07

martin

29

1

This reply describes one way to compare p-values, but the original question seems to remain unanswered: does the procedure make any sense and how does one interpret the results? – whuber Jan 31 '12 at 15:58

What sense does it make to compare p-values to each other?

4 Answers4

Linked