1

What do you think about using a permutation test each time the sample size is too small to meet the the assumptions of normality and homogeneity of variance etc ? Why do we use a non parametric wilcoxon test most of the time?

If there is any, could you please give me the pros of using a permutation test instead of a wilcoxon test ? can we say, since there is many repetitions and since an empirical distribution is generated with the permutation test, that the results are more "trustworthy" in presence of a small size (n=14) ?..

Context: I want to know if there is a significant different between two groups (placebo/treated group) but the distributions don't meet the normality assumption so i have to use a nonparametric test. I heard that permutation test is better than wilcoxon test why n is small but I don't get why? Is it because there is many repetition (permutations of the samples) in the permutation test and in the wilcoxon test there is not, so it give more "reality" to the result? or its just maybe because a permutation test is more powerful since the statistic usually used i just like the t-test ?...

If you have some sources, papers etc feel free to share !

learners
  • 579
  • 6
  • 20
  • Why not t-test? https://stats.stackexchange.com/q/121852/35989 – Tim Nov 03 '22 at 16:47
  • 1
    Presumably because the OP seeks a guarantee on the type I error rate that is absent with a t-test in small samples. – Glen_b Nov 04 '22 at 00:23
  • 1
    @learners A few comments on your post that could lead to clarifications (or relate to a potentially mistaken premise). 1. a permutation test is nonparametric. 2. a Wilcoxon test (by which I presume you mean a WIlcoxon signed rank test here) is a permutation test, performed on the signed ranks. 3. As for why you might prefer a signed rank test or a permutation test using some other statistic, this in part depends on what specific hypothesis you seek to test, (which you make no mention of), and potentially on what alternatives you might particularly look for power against. – Glen_b Nov 04 '22 at 00:28
  • @Glen_b don't understand what you mean by Wilcoxon test is a permutation test performed on the signed ranks? Do we repeat a n number time in the wilcoxon test? is the wilcoxon test based on a empirical distribution too? Sorry I am confused. Also, I don't just mean paired wilcoxon test, I also mean unpaired wilcoxon test. My hypothesis is that I want to know if there is a significant different between two groups (placebo/treated group) but the distributions don't meet the normality assumption so i have to use a nonparametric test. I heard that permutation test is better than wilcoxon but why? – learners Nov 04 '22 at 09:33
  • I edited my post – learners Nov 04 '22 at 09:40
  • Just for clarity in these comments for any future reader. The classic nonparametric test in question is a two-sample unpaired test. The Wilcoxon rank sum test, or Mann-Whitney test. – Sal Mangiafico Nov 04 '22 at 17:25
  • The Wilcoxon-Mann-Whitney test essentially tests if the values in one group tend to be larger in than the values in the other group. In contrast, a permutation test on the means tests if there is a difference in the means. You could also use a permutation test on the medians, or any other statistic of interest. ... I think the real answer to your question is, What hypothesis do you actually want to test ? – Sal Mangiafico Nov 04 '22 at 17:28
  • 1
    @learner I see now you meant either WIlcoxon test (only mentioning one sample size in the Q. threw me off). It sounds like you've got a slight misapprehension about what a permutation test is. A permutation test does not require random sampling of the permutation distribution - with small samples you can determine the complete permutation distribution (the thing you'd be sampling from if you used random sampling to undertake the exchanges of quantity that don't change the distribution under H0). With many rank-based tests you can derive a recurrence relation (simple in the case of ... ctd – Glen_b Nov 04 '22 at 21:22
  • 1
    ctd ... the two WIlcoxon tests) to obtain the exact null distribution at middling sample sizes, and you also know various moments (e.g. the variance) of the test statistic under H0 and can rely on an asymptotic approximation in large samples (normal in the case of Wilcoxon tests). – Glen_b Nov 04 '22 at 21:23

1 Answers1

4

I think the answer to the question as to which test to use comes down to, Which hypothesis do you want to test ?

The Wilcoxon-Mann-Whitney test essentially tests if the values in one group tend to be larger in than the values in the other group.

In contrast, a permutation test on the means tests if there is a difference in the means.

You could also use a permutation test on the medians, or any other statistic of interest.

Start by deciding what you want to know, and then find an approach that addresses this. Note that your description of your goal: "I want to know if there is a significant different between two groups." is vague. There are many ways the values for those two groups could be different. (Mean, median, 90th percentile, ranked values, overall distribution...)

As general advice, be cautious of claims you encounter that you might phrase as, "I heard that permutation test is better than wilcoxon test why n is small". What's the source? Is it reliable? Was there any support or explanation?

Sal Mangiafico
  • 11,330
  • 2
  • 15
  • 35
  • 3
    You could even do a permutation test on the difference in mean rank . . . – Glen_b Nov 07 '22 at 05:40
  • 2
    As an historical point, the Wilcoxon test was much easier to compute than a permutation test, when the Wilcoxon test was developed. It was easy to rank the values and calculate the sum of the ranks and look in the table. Even for modest samples, computing a permutation test was complicated without a computer. There are arguments from nonparametrics about choosing the scores to use, e.g., ranks for the Wilcoxon. One choice of scores gives the permutation test, which is best under certain conditions, possibly most. Permutation tests, now easily computed, could be more widely used. – David Smith Nov 07 '22 at 15:00
  • https://books.google.de/books?id=nvDqDwAAQBAJ&pg=PA132 this book says on page 132 that the wilcoxon rank sum test is giving the same results as a permutation test computed with the rank sum statistic – Ggjj11 Feb 19 '24 at 19:45