9

It strikes me that the available corrections for multiple comparisons in the context of a repeated measures ANOVA are excessively conservative. Is this actually the case? If so, what are some citations I can use to support this point and learn more?

whuber
  • 322,774
russellpierce
  • 18,599
  • What kind of MP correction are you referring too? (BTW, why the alpha tag?) – chl Jul 16 '11 at 20:51
  • 1
    re the alpha tag: Only because the corrections typically are meant to hold alpha constant. – russellpierce Jul 16 '11 at 23:04
  • I'm not sure what MP stands for in this context, but I'm talking about the frequently used Bonferroni and/or Sidak multiple comparisons corrections. – russellpierce Jul 16 '11 at 23:05
  • I know there are other approaches, e.g. False Discovery Rate and so forth, but I am specifically looking to evaluate and critique the more frequently used approaches. – russellpierce Jul 16 '11 at 23:07
  • Sorry, a typo of mine: By "MP correction", I meant MCP (multiple comparisons procedure). I had in mind Tukey HSD (which acts on the test statistic, not the p-value), but your question is interesting (my +1) because it call for a critical appraisal of recent literature on that problem. – chl Jul 17 '11 at 11:06
  • Ah yes. Last time I'd taken a look at this issue I saw that Maxwell & Delaney mentioned a Tukey HSD approach but they also criticized it for ignoring violations of sphericity. – russellpierce Jul 17 '11 at 15:28
  • 1
    As I keep digging around there seem to be essentially two camps... those who have a 'better approach' via a pooled error term and those who want to adjust the p-value according to some sort of multiple comparison procedure (Bonferonni, Holm... etc etc... there seems to be more of them than I have fingers). – russellpierce Jul 17 '11 at 15:30

2 Answers2

5

To the best of my knowledge, the joint distribution of linear contrasts has been derived in the simple ANOVA case (see the documentation of the multcomp R package), but there is are no closed forms for the repeated measures setup. Nevertheless, you can always Bootstrap the joint distribution of these linear contrasts under the null,and look at the minimal t-statistic (or maximal p-value) for setting the significance threshold with FWE control. As you also suggested, you can use methods which only require some qualitative condition on the joint distribution of the test statistics. Bonferroni is a good option if you have few contrasts. Otherwise, have a look at Holm's. If you are looking into many linear contrasts, you should definitely ask yourself you want protection from any false discovery or only a proportion of false discoveries. In the latter case, use the BH procedure for FDR control.

JohnRos
  • 5,684
1

Here's a collection of links to a SPSS forum. Hope you find it relevant to you to some degree: this, this, this, this.

ttnphns
  • 57,480
  • 49
  • 284
  • 501
  • 1
    Thanks for pointing in a direction. Unfortunately news group posts are far from compelling evidence when discussing these issues in published work. The general primer on sphericity is interesting, and points out why Bonferroni-stype p-value adjustments are still in common use. Unfortunately, there isn't really an indictment of the conservative nature of that approach. – russellpierce Jul 17 '11 at 15:34