0

I am extremely rusty at statistics so maybe this is an 'easy' question but I haven't been able to find what I would like to know with a simple search.

I am using now a Wilcoxon signed rank sum test to compare two methods ( classifiers) on the quality (TSS) obtained for each method trained on the same dataset (10-fold cross validation).

I would like now to extent this comparison to more than two methods. Should I now just do a pair-wise comparison for all methods and list this in a table, so for three methods test (A,B) (A,C), (B,C) separately? Or, is there some other method that I should use?

Aktaeon
  • 73
  • Re "Wilcoxon signed rank sum test" .... no, you can have a "Wilcoxon signed rank test" or a "Wilcoxon rank sum test". I presume your data are paired in which case I expect you mean the first of those two. – Glen_b Apr 12 '16 at 16:11
  • Yep you are right, I shouldn't trust blindly the internet ;-) http://www.statstutor.ac.uk/resources/uploaded/wilcoxonsignedranktest.pdf – Aktaeon Apr 12 '16 at 18:06
  • Yes, a certain amount of caution is needed ... but your question still contains the incorrect phrase. You could always look at more than one thing (though if they're written by people in the same area they could both be copying the same wrong thing, so even then caution may be needed). The document is indeed discussing the signed rank test (not that you should simply trust Wikipedia either, but at least it offers some source references). – Glen_b Apr 13 '16 at 00:19

1 Answers1

0

The most commonly used distribution-free test for a multigroup comparison with dependent observations (i.e. akin to the signed rank test* but for more than 2 groups) would be the Friedman test.

* if you're do both tests on two (paired) groups you won't get the same p-values - they're not doing exactly the same thing; the Friedman is the analog of the sign test (though in small samples, if the Friedman won't necessarily give the same answers as the sign test either, since they don't always use a corresponding approximation; one might be exact or use a continuity correction where the other doesn't)

If you're doing cross-validation it's not clear to me that you need a hypothesis test, rather than say a direct comparison of some effect measure, but it may depend on what you're trying to achieve. More explanation of what you're cross-validating and why you're doing hypothesis tests after it would be in order.

Glen_b
  • 282,281