3

I was reading a paper that says they use an "ANOVA permutation test" to test if there is a difference in the (in-)degree distribution among groups of nodes in a network. I Googled this and found a resource (here) on how that works.

In the example in the resource, when they see a significant p-value in the independence_test, they follow up with a pairwisePermutationTest for multiple comparisons. Their results: 

independence_test(Response ~ Factor,
                  data = Data)


Asymptotic General Independence Test

maxT = 3.2251, p-value = 0.005183

So they have a statistically significant p-value here, indicating (I think?) that there is at least one group median that is different from the other group medians.

They follow this up with a pairwise test to determine which medians are significantly different, and obtain these results:

  Comparison    Stat  p.value p.adjust
1  D - A = 0 -0.2409   0.8096  0.80960
2  D - B = 0  -2.074  0.03812  0.06106
3  D - C = 0  -2.776 0.005505  0.01876
4  A - B = 0   1.952  0.05088  0.06106
5  A - C = 0   2.734 0.006253  0.01876
6  B - C = 0   1.952  0.05088  0.06106

As far as I can tell, this implies that the median of group D is significantly smaller than the median of group C, and the median of group A is significantly larger than the median of group C. No other pairwise differences are statistically significant.

Question:

With my data, even though the independence test has a statistically significant p-value, the follow-up pairwise permutation test shows no statistically significant differences. Why could this be?

StatsSorceress
  • 686

1 Answers1

5

This can, and does, happen surprisingly often (in my experience.) It comes about because, heuristically, "almost significant" coefficients are almost as unlikely to come about by chance as "significant" coefficients, so if you have a reasonably high percentage of them, it's very unlikely, in toto, that there's no effect anywhere; it's just hard to pinpoint where. So the omnibus test indicates significance, but none of the pairwise tests do.

To construct a concrete example, assume you have a model with two coefficients that are independent, and that the t-statistics for them are both equal to 1.8, with, say, 100 degrees of freedom (so we can work with Normal distributions with a reasonable degree of accuracy.) The probability that the sum of the squares of the two t-statistics exceeds 6.48 (=$2\cdot 1.8^2)$ is about 0.04 (under the null hypothesis), so, jointly, the two coefficients are significant at the 95% level of confidence, but obviously neither one is significant at the 95% level of confidence all on its own. The following graph illustrates this:

enter image description here

This may seem a rather unlikely scenario, but as the number of variables / coefficients increase, there is more opportunity for it to happen.

This topic is closely related to the multiple comparisons problem; for more, see Wikipedia.

Glen_b
  • 282,281
jbowman
  • 38,614
  • Brilliant! Thank you for the clear answer and illustrative example. Very helpful. – StatsSorceress Aug 30 '18 at 17:00
  • How are you computing the joint p-value there? – Glen_b Aug 31 '18 at 02:01
  • @Glen_b - $\chi^2(2)$. – jbowman Aug 31 '18 at 14:56
  • The reason I asked is the $5%$ $\chi^2_2$ critical value is $5.99$ (which exceeds $1.96^2 = 3.84$); so the circle should have a larger radius ($\sim 2.45$) -- it should extend beyond the $(-1.96,1.96)\times(-1.96,1.96)$ square near the axes (while being inside it near the diagonal). (i.e. the drawing doesn't correctly represent the bivariate rejection boundary you are discussing). I agree with the gist of the answer and you clearly understand how the bivariate rejection rule works, but I hesitate to upvote because of the issue with the diagram. – Glen_b Aug 31 '18 at 23:36
  • @Glen_b - You're quite correct, of course. I did it in PowerPoint and didn't think through exactly where the circle should lie. Now that I have access to a computer with R on it, I can make a more accurate graph. (I probably could have in Excel too, but I use Excel about once a year and have almost completely forgotten how to.) Then again, the answer is really about the concept of the "In this region" part, but I'll fix it up in a day or so either way. – jbowman Sep 01 '18 at 01:46
  • @jbowman .... would this one do? https://i.stack.imgur.com/LYsjr.png (edit as at 12 Sept: Given the passage of time, I took the liberty of including it in the answer) – Glen_b Sep 06 '18 at 00:48