28

If the model does not satisfy ANOVA assumptions (normality in particular), if one-way, Kruskal-Wallis non-parametric test is recommended. But, what if you have multiple factors?

kjetil b halvorsen
  • 77,844
user4267
  • 341

4 Answers4

19

You can use a permutation test.

Form your hypothesis as a full and reduced model test and using the original data compute the F-statistic for the full and reduced model test (or another stat of interest).

Now compute the fitted values and residuals for the reduced model, then randomly permute the residuals and add them back to the fitted values, now do the full and reduced test on the permuted dataset and save the F-statistic (or other). Repeate this many times (like 1999).

The p-value is then the proportion of the statistics that are greater than or equal to the original statistic.

This can be used to test interactions or groups of terms including interactions.

Greg Snow
  • 51,722
  • 6
    For a discussion of different permutation strategies in factorial ANOVA-designs, see e.g. http://avesbiodiv.mncn.csic.es/estadistica/permut1.pdf (pdf) – caracal Jun 21 '11 at 11:40
  • 4
    This works, but what happens to the power of the test? E.g., even if there is only one (far) outlying value and the rest of the residuals are normally distributed, it appears that using the F-statistic may have little power in the permutation test to detect anything. The paper referenced by @caracal discusses the subtleties and assesses when the F-statistic approach works and when it might fail. – whuber Jun 21 '11 at 14:31
  • "The p-value is then the proportion of the statistics that are greater than or equal to the original statistic" --> to the original statistic calculated on the full model. correct? – Yannick Wurm Nov 08 '11 at 02:57
  • @Greg, does joining the two factors into one to create a new factor, e.g. in R interaction(factor1, factor2), make sense for this situation? JeremyMiles stated here that "The problem you have in analyzing these data is that interactions don't really make sense when you have non-parametric tests. Non-parametric tests consider data to be ranks..." – toto_tico Nov 13 '15 at 05:02
  • 1
    @toto_tico, using ranks is one option for non-parametric tests, but is not the only one (permutation testing is another that does not rely on ranks). Combining factors into a single factor works if you want to test all-or-nothing, but does not work for testing if the interaction is significant beyond the effects of the main effects, or testing one factor given the other factor is in the model. – Greg Snow Nov 13 '15 at 16:57
  • @GregSnow, thanks a lot. Permutation tests seem to be the alternative. The only two packages that I found that accept interactions in the formula are the ez (ezPerm function) and lmPerm (aovp). However, the ezPerm documentation indicates that only the main effects should be trusted, and the aovp produces unstable p-values(see Henrik answer). Any ideas of what can I use instead? – toto_tico Nov 13 '15 at 22:18
  • 1
    @toto_tico, just code it directly. See the example I added based on your other comment (http://stats.stackexchange.com/questions/41199/non-parametric-for-two-way-anova-3x3/41207?noredirect=1#comment344933_41207). – Greg Snow Nov 16 '15 at 17:33
16

The Kruskal-Wallis test is a special case of the proportional odds model. You can use the proportional odds model to model multiple factors, adjust for covariates, etc.

Frank Harrell
  • 91,879
  • 6
  • 178
  • 397
  • 7
    If one would like to learn more about the connection between K-W and the proportional odds model, what would be a good reference? – whuber Jun 21 '11 at 14:24
  • 8
    @ARTICLE{pet89ord, author = {Peterson, Bercedis}, year = 1989, title = {Re: {Ordinal} regression models for epidemiologic data}, journal = Am J Epi, volume = 129, pages = {745-748}, annote = {proportional odds model; partial proportional odds} } @ARTICLE{mcc80reg, author = {{McCullagh}, Peter}, year = 1980, title = {Regression models for ordinal data}, journal = JRSSB, volume = 42, pages = {109-142}, annote = {ordinal logistic model} }See also Whitehead Stat in Med 1993 p. 2257 – Frank Harrell Jun 22 '11 at 02:36
10

Friedman's test provides a non-parametric equivalent to a one-way ANOVA with a blocking factor, but can't do anything more complex than this.

Freya Harrison
  • 3,412
2

One nonparametric test for a two-way factorial design is the Scheirer–Ray–Hare test. It is described by Sokal and Rohlf (1995), and can be found on a variety of websites, though it appears to be not particularly well known or widely discussed.

Another approach is aligned ranks transformation anova (ART anova). With current software implementations, this approach is easy to use, and in some implementations it can handle relatively complex designs including random effects.

References

Sokal, R.R. and F.J. Rohlf. 1995. Biometry, 3rd ed. W.H. Freeman. New York.

Sal Mangiafico
  • 11,330
  • 2
  • 15
  • 35