4

I've conducted a four-phase experiment with physiological measures on two groups (Controls vs. Patients). After applying a Mann-Whitney test to examine differences between groups for each dependent variable at each phase, I found some significant results.

However, when I attempted to build a linear mixed model (LMM) in R using the formula model <- lme(Dep_Variable ~ TIME * Group, random = ~ 1|ID , data= data), the LMM results are substantially different from the Mann-Whitney test. Specifically, significant differences observed with Mann-Whitney at certain time points are not replicated in the LMM, and vice versa.

Are there specific considerations in interpreting results obtained from a Linear Mixed Model (LMM) compared to non-parametric tests? What factors could contribute to differences between Mann-Whitney and LMM results?

Ed9012
  • 311

2 Answers2

8

Mann-Whitney tests "the null hypothesis that, for randomly selected values X and Y from two populations, the probability of X being greater than Y is equal to the probability of Y being greater than X". You probably use your LME model to test hypotheses about explained variance, or about parameters being zero. These are very different things, so the answers that different tests give to different questions will also differ.

Best to first formulate your research question, then choose an appropriate test.

Stephan Kolassa
  • 123,354
  • The thing is that, as a first step, I would like to see if there are any differences between the two groups for each phase of the experiment. Subsequently, I would look into those differences to see which group has higher values. That's the purpose of building a model with all the four phases, and where I found results to be contrasting – Ed9012 Dec 10 '23 at 20:41
  • 2
    You will need to be more specific than "any differences between the two groups". Two groups can have the same mean (which is what t-tests test), but if you draw a random sample from each group, it may still be more likely that the observation from group A is greater than that from group B (which is what Mann-Whitney tests). Or the other way around. If you just want to test whether the CDFs differ, then a two-sample Kolmogorov-Smirnov test may be appropriate. Yes, this is complicated, and there is a lot of imprecise and misleading information out there. – Stephan Kolassa Dec 11 '23 at 07:03
6

In addition to the differences between Mann-Whitney and mixed linear models, as noted by Stephan Kolassa, part of your problem might be:

After applying a Mann-Whitney test to examine differences between groups for each dependent variable at each phase, I found some significant results.

That sounds like you did multiple comparisons over multiple outcome variables at multiple experimental phases. That leads to a potential multiple-comparison problem: if you do enough comparisons, some will pass a given threshold of "significance" even if there aren't any true differences. If you didn't correct the Mann-Whitney tests for the multiple comparisons you might well have found nominally "significant" differences that aren't real.

Note that the Mann-Whitney test can be considered a special case of ordinal logistic regression, a method that allows you to avoid the multiple pairwise testing you thought that you needed to perform. See Chapter 13 of Frank Harrell's Regression Modeling Strategies.

Finally, if a parametric linear model is appropriate then these data might not require mixed modeling. It sounds as if a classic multivariate (multi-outcome) linear model might be OK, or other approaches like those described in Chapter 7 of Regression Modeling Strategies.

EdM
  • 92,183
  • 10
  • 92
  • 267
  • after I correct for FDR or Bonferroni, none of the results are significant. However, the reason why I would use a linear mixed model is to look at the effect of time between the two groups over each dependent variable (so that i build many different models with the same predictors). I don't understand your suggestion on how the MW test can be used to avoid the multiple pairwise testing, could you please write more details? – Ed9012 Dec 07 '23 at 21:16
  • 1
    @Ed9012 what I meant to convey is that ordinal logistic regression, which allows for complex linear multiple regression models without assumptions about error distributions, can be used instead of MW, which can only compare two groups at a time. See Chapter 13 of the Harrell reference and this resource. If you have complete data, multivariate linear models might provide an alternative to mixed models. That said, there's nothing necessarily wrong with your mixed models; if the assumptions are met I'd trust those results. – EdM Dec 07 '23 at 22:38
  • I did not mention though that my dependent variables are continuos. I see how I can use multivariate linear models, building models like the following: model<-lm(Dep_Variable~Group,data=TIME1). In this way I can still see if there are any differences between the group for each phase for multiple variables, though I need to split the original dataset several times. Does that sounds right? – Ed9012 Dec 07 '23 at 23:38
  • I mean, that would still be a simple linear regression... I don't see how I can apply the techniques you mentioned. I did Mann-Whitney because the data are not normally distributed. – Ed9012 Dec 07 '23 at 23:53
  • @Ed9012 you typically should not do separate analyses for each time point/phase. That tends to have less power than a comprehensive model of all the data. The usual normality assumption in multiple linear regression is about the error distribution estimated by residuals between observations and model predictions, not the raw data. Even that assumption can sometimes be relaxed. See this answer. Ordinal regression doesn't even need that assumption; it just works with the rank-order of observations as Mann-Whitney does. – EdM Dec 08 '23 at 15:24