Assumptions of Mann-Whitney test for at least ordinal data

Question

I am reading article by Divine et al. about using Mann-Whitney test for data that is at least ordinal (i.e. it may be discrete with many ties). It says the following (in section 2.3):

That is, it (the Mann-Whitney test) generally does not depend upon any particular distributional form (or parameters) in order to generate the test statistic and p-value. In fact, it is the whole distributions that are being compared, rather than any sample-specific summary statistic(s). However, the procedure does depend upon some assumptions about those distributions. For instance, one important assumption is that the variances of the two distributions should be the same (Pratt 1964).

And in section 5.1 this paper recommends to use the Brunner-Munzel test instead of the Mann-Whitney test if the variances are unequal (as well as scipy.stats.brunnermunzel manual):

Although the basic WMW test may be invalid with unequal variances (especially with unequal sample sizes), the Brunner–Munzel variation should work if the minimum sample size is at least 30 and the variance discordance is not too extreme. For a sample size (or sizes) below 30 and/or when one or more large clumps of ties are present, an exact/permutation WMW test (available in SAS and R) should be considered.

The hypotheses in this article are formulated as follows (in the two-sided alternative case; $X_1 \sim F, X_2 \sim G$):

• $H_0: ~ P(X_1 \gt X_2) + \frac{1}{2} P(X_1 = X_2) = \frac{1}{2}.$
• $H_1: ~ P(X_1 \gt X_2) + \frac{1}{2} P(X_1 = X_2) \neq \frac{1}{2}.$

I am wondering what are the other assumptions of such Mann-Whitney test? (besides equality of variances and independence of samples; if we want to use this test for some at least ordinal data, i.e. not necessarily continuous)

---

In the famous article by Fay and Proschan (2010) there is a very similar formalization (perspective) of the Mann-Whitney test which is given for continuous data:

where $\Psi_C$ is the set of all continuous distributions, $H_3$ is the null and $K_3$ is the alternative, $\mathrm{P} = H_3 \sqcup K_3$ is the full set of allowed distributions.

The assumption of equal variances (which I mentioned earlier, see the beginning of this post) is one of the requirements which are introduced to guarantee that $\mathrm{P}$ will not contain distributions with both $\phi(F,G) = 1/2$ and $F \neq G$. And I want to know what are the other assumptions (besides equality of variances) we need to guarantee that.
Indeed, according to article by Karch (2021), "The assumptions for the different perspectives are all a special case of the Mann-Whitney test’s core assumption, exchangeability. In the Mann-Whitney test setting, exchangeability reduces to if the null hypothesis is true, the two population distributions must be identical." In other words, different perspectives have different null hypotheses but in each case the full set of the allowed distributions $\mathrm{P}$ shouldn't contain distributions $(F,G)$ for which it is possible to have $F \neq G$ under the null. That's why for each perspective we have different set of assumptions (i.e. restrictions on $\mathrm{P}$) to guarantee that.

Fay and Proschan require continuous distributions here (although they defined $\phi(F,G)$ both for discrete and continuous distributions). I guess that they require this because the consistency of Mann-Whitney test is strictly proved only for continuous distributions. However, the article by Divine et al. shows that the aforementioned formalization of Mann-Whitney test (it is given at the beginning of my post as well as hyperlink to the article) is perfectly valid for discrete data (which possibly contain many ties).

Answer 1

The null hypothesis of the MW-test, under which the distribution of the test statistic is computed, is that $H_0:\ F=G$, the two distributions are the same. This obviously implies that their variances are the same, but the latter "assumption" doesn't actually add anything (see below though). It is also assumed that data are i.i.d.

I think the confusion about ties comes from imprecision of what is actually meant when referring to the MW-test, just the test statistic, or also the distribution under the $H_0$. If there are ties, both asymptotically and for finite samples, the distribution under the $H_0$ that is used for testing has to be modified. This can be done (so the test can be applied), however the test can be seen as invalid if this is not done.

Now how about the "equal variances" assumption? I have mentioned the null hypothesis, however one can state that a valid test does not only require that the distribution under $H_0$ is correctly specified, but also that it has some properties under the alternative. Something of a minimal requirement is that the test should be unbiased, i.e., that the probability to reject under any distribution in the alternative should not be smaller than $\alpha$, the probability to reject under the $H_0$. Unbiasedness follows easily for the alternative that I have learnt (and that is one of the possibilities mentioned in Fay and Proschan), which is that $F$ is stochastically larger than $G$ (i.e., the cdf of $F$ is everywhere smaller or equal than that if $G$, and somewhere smaller). This does not require equal variances, and neither does "Perspective 3" as cited above from Fay and Proschan. Although there are examples of pairs of distributions with unequal variances with $F\neq G$ and $P(X_1>X_2)+\frac{1}{2}P(X_1=X_2)=\frac{1}{2}$ (I believe though I haven't checked that this holds for two Gaussian distributions with equal mean and different variances), I don't think it makes sense to say that the MW-test "assumes equal variances". Computation of the distribution of the test statistic under $H_0$ assumes even more than that, and the valid alternatives stated above against which the test in unbiased contain many pairs of distributions with unequal variances.

In fact one could state that using the first Alternative given in the question (which amounts to Fay and Proschan's Perspective 3) there is no further assumption beyond i.i.d. at all, as this contains all distributions. But Julian Karch (see his answer) has shown that the MW-test is not generally unbiased against this alternative. If you are really interested in this alternative, he recommends the Brunner-Munzel test.

However there may be assumptions implied by certain interpretations that are given to the test result, so this is something to be careful about. If for example a rejection of the null hypothesis is taken as evidence that $F$ is stochastically larger than $G$, one should know that the test also is unbiased against some alternatives for which this isn't the case, and it is implicitly assumed that these do not obtain (one such possibility would be Gaussian distributions with different means and different variances - this belongs to the "Perspective 3" alternative as far as I can see, but not to the "stochastically larger" alternative). Also, as Fay and Proschan mention, there are distributions for which $F\neq G$ and $P(X_1>X_2)+\frac{1}{2}P(X_1=X_2)=\frac{1}{2}$, which cannot be detected by the MW-test (although it is not so clear whether the user in such a case rather would want to reject, or whether they'd be happy to say that there is no evidence that one distribution tends to be larger than the other).

The MW-test can be safely used to test $F=G$ against the "stochastically larger"-alternative, which is how I think most people would interpret the test result, i.e., $F$ tends to produce systematically larger (or smaller) observations than $G$. The issue here is that not everything that is possible is covered, i.e., in reality it may be the case that $F\neq G$ but none of them is stochastically larger than the other, for example $F$ may produce more very large and more very small observations than $G$. In a real application I'd therefore look at visualisation such as boxplots and histograms to see whether this might be the case, and interpret results with caution.

Summarising, Fay and Proschan's distinction of different "perspectives" is important, because in fact different perspectives make different implicit assumptions when interpreting the test result, and not being aware of this may lead to misinterpretation. One could say that running the test itself, mathematically, does not require such assumptions (one can just take as null hypothesis all distributions that have rejection probability $\le\alpha$ and as alternative all those for which the rejection probability is larger), but making sense of the result does.

Answer 2

I just stumbled across this, and since I am the author of Karch (2021) and do not fully agree with the answers so far, here are my two cents. I will skip the assumption of no ties as there is agreement that it is unnecessary (for the alternatives Christian and I discuss).

We have to first decide what properties the assumptions should guarantee. Fay and Proschan (2010) and I (influenced by them) focussed on [approximate] validity (type I error rate is below significance level $\alpha$ [at least in large samples]) and consistency (with larger samples sizes power approaches 1). We also have to agree on what the proper alternative is. I agree with Divine et al. that it should be $H_1:p\neq\frac{1}{2}$, with $p=P(X<Y) + \frac{1}{2}P(X=Y)$. I am surprised that there is controversy around this since the test statistic used is the sample equivalent of $p$ (see Karch (2021), p. 6).

Under this setup, the Wilcoxon-Mann-Whitney (WMW) test requires that $H_0:F=G$ is used as null hypothesis (see Fay and Proschan (2010), Table 1). Rephrased as assumption, we thus have to be sure that if $F$ and $G$ are not equal, $p\neq \frac{1}{2}$.

Fay and Proschan call this Perspective 3 and state that this situation is unrealistic (This is already in the question, but I felt it was important to highlight this), with which I fully agree. To make this quote understandable, I define $\mathcal{M}:=H_0\lor H_1$. Note that I changed the notation slightly.

... Perspective 3 ... is a focusing one since the full probability set, $\mathcal{M}$ is created more for mathematical necessity than by any scientific justification for modeling the data, which in this case does not include distributions with both $p = 1/2$ and $F \neq G$. It is hard to imagine a situation where this complete set of allowable models, $\mathcal{M}$, and only that set of models is justified scientifically;

Thus, while this is technically the correct assumption for the WMW it is hard to imagine situations in which it is actually met and thus a bit irrelevant. One example that is outside of $\mathcal{M}$ is that $F$ and $G$ are normal but have different variances. I demonstrate in Karch (2021) that type I error rates of the WMW test can be inflated in this example, even in large samples.

Beyond this, if we extend the properties our assumptions should guarantee to be correct standard errors, good power, and confidence intervals with correct coverages, which seems reasonable, then the WMW is not appropriate even under the unrealistic Perspective 3. As Wilcox (2017) says:

A practical concern is that if groups differ, then under general circumstances the wrong standard error is being used by the Wilcoxon–Mann–Whitney test, which can result in relatively poor power and an unsatisfactory confidence interval. (p. 279)

To give an example consider $F=\mathcal{N}(0, 2)$ and $G=\mathcal{N}(0.2, 1)$. The alternative hypothesis $H_1$ is thus true. However, the WMW test can be biased in this situation (the power is smaller than the significance level $\alpha$). See:

set.seed(123)
library(brunnermunzel)
reps <- 10^3
p_wmw<- p_BM <- rep(NA, reps)
for(i in 1:reps){
  g1 <- rnorm(80, mean = 0, sd = 2)
  g2 <- rnorm(20, mean = .2, sd = 1)
  p_wmw[i] <- wilcox.test(g1, g2)$p.value
  p_BM[i] <- brunnermunzel.test(g1, g2)$p.value
}
print(mean(p_wmw < .05))
[1] 0.034

Overall, the situation is equivalent to the much more well-known and appreciated problems with Stundent's $t$ test. Again from Wilcox (2017):

The situation is similar to Student’s T test. When the two distributions are identical, a correct estimate of the standard error is being used. But otherwise, under general conditions, an incorrect estimate is being used, which results in practical concerns, in terms of both Type I errors and power. (p. 278)

Just as Welch's $t$ test is a small modification of Student's $t$ test that alleviates these problems, as it provides correct standard errors in general circumstances, Brunner-Munzel's test is a small modification of Wilcoxon's test that provides correct standard errors in general circumstances (both tests can still fail in smaller samples, but problems are much less severe, as at least asymptotically Brunner-Munzel's test provide correct standard errors). There seems to be widespread agreement to use Welch's instead of Student's t test for these reasons (see, for example, Is variance homogeneity check necessary before t-test?). For the same reasons, we should usually use Brunner-Munzel's instead of Wilcoxon's test.

The assumptions for Brunner-Munzel's test to have correct standard errors in large samples are rather general and technical. They are described in detail in Brunner et al. (2018). However, they are so general that they are rarely violated. A more practically relevant question is what sample sizes are needed in practice for the standard error to be "correct enough". Simulation studies (see Karch (2021), as well as the reference therein) suggest that this is true for rather small sample sizes. No meaningful type I error inflation have been found yet for $n_1,n_2\geq 10$. However, for small samples sizes the permutation version of the test is recommended.

Thus, in practice, it seems fine to treat the Brunner-Munzel test as test for $H_0:p=\frac{1}{2}, H_1:p\neq\frac{1}{2}$, without additional assumptions (beyond i.i.d). As all the problems of the WMW test just discussed tend to disappear for equal samples (see, Brunner et al. (2018); note that this is again equivalent to Student's t test) it also seems fine use the WMW instead when sample sizes are (roughly) equal. I would still use the Brunner-Munzel test even if sample sizes are equal as it's implementations in R provide confidence intervals for $p$, whereas the WMW implementations (I am aware of) do not.

Answer 3

There is some disagreement as to the 'proper' use of the two-sample Wilcoxon (rank sum) test. Perhaps this is because it is often used in ways that might surprise its creators and because various software programs have implemented a wide variety of versions to accommodate (moderate proportions of) ties and other departures from canonical assumptions.

One way to be reasonably sure how the Wilcoxon RS test works in a particular situation is to try it out and see what actually happens.

The following brief simulations address the assumption that the two populations must be of the same shape, differing only by a shift; this assumption is often taken to mean that the population variances must be equal.

By contrast, the implementation in R can be viewed as a test whether one distribution stochastically dominates the other--up to a point, regardless of shape or of variance.

I use the test to compare samples of size 50 from distributions (a) $\mathsf{Norm}(\mu=100,\sigma=5),$ (b) $\mathsf{Norm}(\mu=100,\sigma=10),$ and (c) $\mathsf{Norm}(\mu=105,\sigma=10).$

First, we use the Wilcoxon SR test to compare null (a) with alternative (b), a difference in shapes; second, to compare null (a) with alternative (c), a difference in shapes and locations.

set.seed(1123)
pv = replicate(10^4, wilcox.test(rnorm(50, 100, 5), 
                      rnorm(50,100,10))$p.val)
mean(pv <= .05)
[1] 0.0577         # (a vs b) true level about 6%, not exactly 5%
par(mfrow=c(1,3))
 hist(pv, prob=T, col="skyblue2", main="Same Centers")
pv = replicate(10^4, wilcox.test(rnorm(50, 100, 5), 
                      rnorm(50,105,10))$p.val)
mean(pv <= .05)
[1] 0.8483         # (a vs c) power about 85%
hist(pv, prob=T, br=20, col="skyblue2", main="Different Centers")
curve(pnorm(x,100,5),50,150, lwd=2, col="green3", lty="dashed")
 curve(pnorm(x,100,10), add=T, col="blue")
 curve(pnorm(x,105,10), add=T, col="maroon", lty="dotted")
par(mfrow=c(1,1))

The first panel of the figure shows the roughly uniform distribution of of P-values of comparison (a) vs (b), and the second shows the power (left-most histogarm bar) of comparison (a) vs (c).

The third panel shows that neither distribution (a) [broken green] nor (b) [solid blue] is stochastically dominant. It also shows that (c) [dotted red] dominates (a), plotting mainly to the right of and below (a).

Finally, we note that, because data are normal, the most appropriate test to compare (a) and (b) would be a two-sample Welch t test, which does not assume equal variances; its significance level is very near the nominal 5% level (no figure).

set.seed(1123)
pv = replicate( 10^4, t.test( rnorm(50, 100, 5), 
                       rnorm(50,100,10) )$p.val )
mean(pv <= .05)
[1] 0.0484      # aprx 5%

The point here is not to give an exhaustive catalog of the properties of any one implementation of the Wilcoxon RS test. It is to illustrate how simple simulations can help to settle particular controversies.

Note: Original versions of the Wilcoxon rank sum test and the Mann-Whitney U test used different, but essentially equivalent, test statistics.

Addendum, per Comment. If the task is to test whether $\mathsf{Beta}(1,3) \ne \mathsf{Beta}(3,1),$ based on ten observations from each distribution, then the two-sample Wilcoxon test (2-sided) will do the job with power very nearly 1:

set.seed(2022)
pv = replicate(10^5, wilcox.test(rbeta(10, 1,3), rbeta(10, 3,1))$p.val)
mean(pv <= 0.05)
[1] 0.99692

However, it seems that the meaning of rejection ('perspective') should not be that the median of the former distribution is about $\eta_1=0.2063$ and $\eta_2=0.7937,$ and even less that the median has "shifted" upward. The two distributions have very different shapes.

It is clear from plots of empirical CDF of two samples of size ten that $\mathsf{Beta}(3,1)$ (blue) dominates (tends to give larger values than) the former:

 set.seed(622)
 x1 = rbeta(10, 1, 3)
 x2 = rbeta(10, 3, 1)
hdr="ECDF Plots: BETA(3,1) Dominates"
 plot(ecdf(x2), col="blue", xlim=0:1, main=hdr)
 plot(ecdf(x1), add=T, col="brown")