Wilcoxon Signed Rank

Question

I have 4 matrices of size 686810 each. To understand better, we can represent each matrix as a graph of 68 nodes.

For each node I compute betweeness centrality, strength, clustering coefficient, and vulnerability. So, for each parameter for example betwennes centrality I have a matrix of 10*68, and the same for the rest of the 3 other measures.

My request is:

I have to apply wilcoxon signrank test ( 68*4 multiple comparisons) for each measure of each node. The value of the measure of a node is considered significant if the test showed that the values of the measure were significantly higher than the median of that measure across all nodes. The level of significance was padjusted < 0.05.

So how can I apply it in this case?

Answer 1

You can use the Wilcoxan Signed Rank test, but you want to control the family-wise error rate by adjusting your significance level for the individual tests. This XKCD is actually an accessible introduction to multiple comparisons testing that doesn't control for the family-wise error rate. Basically, with that many tests and a p-value of 0.05 for each test, you will get a lot of "significant" results (somewhere around 10-15*) even if all of your matrices are really the same.

My recommended correction for a case like this is to do the means comparisons with a smaller p-value, calculated to achieve the desired False Discovery Rate (original paper here).

False Discovery Rate calculations are supported in R. I prefer this one.

*5 out of 100, or about 5% of the tests.