2

I am comparing three distributions of values. Let's say A, B and C.

If I find that the p-value of the Wilcoxon rank sum test between A and B is 1e-08 and between A and C is 1e-10, can I say that distribution A is more similar to B than it is to C?

mdewey
  • 17,806
panos
  • 25
  • 3
  • Given those values, I don't think there is much meaning left in saying A is more similar to either one. – Marc Claesen Sep 18 '13 at 16:44
  • 'Wilcoxon rank-sum test'? – Scortchi - Reinstate Monica Sep 18 '13 at 16:44
  • 1
    Scortchi, yes it's a Wilcoxon rank-sum test. – panos Sep 19 '13 at 06:59
  • 1
    Glen_b, I'm using it in order to compare if distributions of values differ from each other or not. More specifically, I'm studying the length of introns between different organisms and want to see if the introns from one organism are significantly longer/shorter compared to another. – panos Sep 19 '13 at 07:05
  • Useful information. It might be worth adding that into your question. – Glen_b Sep 19 '13 at 07:28
  • 1
    I don't feel completely comfortable with the use of the term "distribution" in the question. Most inferential tests (like Wilcoxon) compares the central tendency, not the shape of the distribution. We're dealing with ordinal type data, so it is probably not as important to consider this. Still, for example, B and C could have the exact same median but B could be bimodal and C could be unimodal. Something to think about? – Hotaka Sep 19 '13 at 15:26
  • @HotakaMaeda Many thanks for the comment. What do you suggest using if I want to compare the actual distributions? Maybe ANOVA? – panos Sep 19 '13 at 19:05
  • It really depends on what you're trying to do with the data, and tests to compare distributions aren't something I'm too familiar about, but plotting the data should always be the first step. Visualizing the data can tell you things that numbers cant, especially about the distribution. Use side-by-side boxplots or histograms. Have you done that yet? – Hotaka Sep 19 '13 at 19:23
  • Yes, I've also done boxplots, which agree with what I get from Wilcoxon. I'm just curious what my alternatives would be on the statistical test part. Thanks for the input! – panos Sep 19 '13 at 20:36

2 Answers2

7

No. According to Gelman and Stern (2012), "even large changes in significance levels can correspond to small, nonsignificant changes in the underlying quantities." In plain English, differences between p-values are generally not statistically significant.

zkurtz
  • 2,160
1

No. To say the least, it would depend on sample size. More importantly, why would you want to do that? I'd simply do 3 tests to compare all 3 with each other using Bonferroni correction or similar.

But if you're dealing with length, why not ANOVA??

Hotaka
  • 1,174
  • 9
  • 14
  • 1
    I believe it's not possible to get p-values that low on the Wilcoxon test unless the sample sizes are pretty large (or at least of moderate size--$10$ or greater--and the differences are as extreme as possible). But given that the differences are stated in terms of p-values and not effect sizes, why and how would any such conclusion depend on sample size? – whuber Sep 19 '13 at 07:29
  • Well, the test is a between-subjects comparison, so A, B, and C could have differing sample sizes. Looking at the p-values, it may seem like A and C are further apart than A and B, but it might just mean they are both the same distance and C just has a higher sample size. – Hotaka Sep 19 '13 at 15:17