6

I have a 5 x 2 design where one of the levels of the first factor is a control, all others are experimental conditions. I'm interested in the interaction between the two factors and especially, if the interaction is present for each of the 4 experimental conditions with the control condition.

I conduct 4 separate 2 x 2 ANOVAs where I pair each experimental condition with the control condition in the first factor. This seems to call for p-value adjustment to avoid the multiple-testing problem but what do I have to adjust? Are all p-values adjusted (the two main effects and the interaction)? Or do I only adjust the interaction p-values? Or separately for the main effects?

Thanks a lot

whuber
  • 322,774
thias
  • 965

2 Answers2

1

If you're just focusing on the four interactions, I would adjust, but only for those 4 tests.

I think most people wouldn't worry about the multiplicity adjustment here; I seldom see such adjustments made unless the number of tests is quite large.

Karl
  • 6,197
  • I would not mind to adjust but it was required by a reviewer. – thias Sep 30 '11 at 12:33
  • 2
    @thias - In responding to reviewers' requests on a paper, I'll do whatever they suggest unless I think it's completely wrong, and even then I'll make some modification to the paper in response. – Karl Sep 30 '11 at 13:01
  • 2
    Good answer, Karl, but could you explain why it's valid to adjust only for the four interaction tests? There also appears to be a complication because the four ANOVAs clearly are dependent: they share the control data. It's unclear then how to adjust the p-values. Shouldn't this testing be done with a single ANOVA, protected by an overall F test, followed by a set of four contrasts to test the interactions? – whuber Sep 30 '11 at 16:52
  • 1
    @whuber - [Oog; too many important questions.] I'd adjust only for the tests I really cared about (hoping that the reader would trust that I really did only care about those), but of course it's important that what I care about was determined in advance and not after looking at the results. I agree that it's not clear exactly how to adjust, given the dependence. I interpreted his question as that he had done the overall test and was now looking at the individual pieces; it does seem in looking at those pieces that you should account for the fact that there are four of them. – Karl Sep 30 '11 at 17:11
1

p-value adjustments

P-value adjustments are often designed to control Type I error rates for a set of analyses. There are conventions regarding what is often conceptualised as a set and what is not conceptualised as a set, but such conventions should not be taken too seriously. For example, your four interaction effects of interest might be seen as a set.

Or try to increase parsimony of analyses

Alternatively, you could try to perform your hypothesis testing in a different way in order to minimise the multiplicities in your analysis, or make some analyses conditional on success of previous analyses.

For instance, you could do the following:

  1. First, perform a compound comparison which defines factor 1 as either experimental or control and then tests for the interaction with factor 2. This will tell you in general whether the experimental conditions have a different effect of factor 2 than control.
  2. If the previous compound comparison is significant, you could then do a separate ANOVA (4 x 2) that excludes the control group and thus tests whether there are any differential effects of factor 2 by experimental conditions.
  3. If the previous interaction effect was significant, you could perform some test of which effects in experimental conditions were larger than others (perhaps Tukey's HSD on factor 2 change score).
Jeromy Anglim
  • 44,984
  • thanks, that seems valid. Actually, that is what I did: I conducted the 5x2 ANOVA first and proceeded to checking the separate 2x2 ANOVAs. However, I also need to report and interprete the main effects of the four ANOVAs. Would I have to adjust across all main effects and interactions together? Or can I have separate sets of adjustments for separate hypotheses (the hypotheses concerning the main effects are largely separate from the interaction hypotheses)...? – thias Oct 03 '11 at 08:20