What if an overall ANOVA is not significant, but specific contrasts are?

Question

Study Design: I have a 2x3 factorial design, 2 levels of Time (2050 or 2100) by 3 levels of information (None/Control, Moderate, Extreme).

I set up some very specific contrasts when analyzing this design, in particular Control vs. Other, Moderate vs. Extreme, and 2050 vs. 2100. For one DV, the overall ANOVA was not significant but the Control vs. Other contrast was.

Question: What is the best way to interpret a nonsignificant ANOVA but a significant contrast?

I know that ANOVA is used to lower familywise error rates, and I wouldn't want to fall into the trap of ignoring the overall ANOVA for pairwise comparisons. However, this is one of a select few contrasts that I planned ahead of time. Does that make a difference in interpretation, or should I simply consider the manipulation not significant?

There are a lot of posts on this site that ask, essentially, this same question. Here is one example: possible duplicate of How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure? — Macro, Jun 08 '12 at 01:59
Good catch, Macro. The present question, however, appears to focus on a different aspect of this issue: interpretation. Given that the replies to the duplicate do not appear to address this at all, I am reluctant to vote to close. — whuber, Jun 08 '12 at 12:51
Also, the question you suggest as a duplicate is something of an "opposite" -- it asks what to do when ANOVA is significant but post-hocs are not. In this case I was asking how to interpret it when the ANOVA is non-significant but the post-hocs are. — Lee, Jun 15 '12 at 20:59

score 10 · Answer 1 · answered May 21 '12 at 14:36

10

The answer to your question reflects your view of overall (or experimentwise) alpha. If, before you collected your data, you set forth planned comparisons (and it sounds like you did that), then after you collect the data you do those and only those comparisons, and there is no reason to look at any other comparisons (AKA contrasts) nor at the overall ANOVA. Why did you look at the overall ANOVA? If you did both planned comparisons and ad hoc comparisons, then you inflated your experimentwise alpha, which some consider a serious error. Of course, you might do ad hoc comparisons to help you plan your future research, but not report them. Also, it seems you have two DVs and did two ANOVAs. In general, if you do two ANOVAs each with p = .05 then you have inflated your experimentwise alpha. Therefore, when doing 2 ANOVAs you might reduce the alpha level for each of your two ANOVAs to maintain your experimentwise alpha. Alternatively, you might do a MANOVA (i.e., a multivariate analysis of variance) which allows you to consider both DVs simultaneously. Of course, you should also be looking at effect sizes, to help you understand the meaning of your data.

answered May 21 '12 at 14:36

Joel W.

3,306

Perhaps I am mistaken, but I was under the impression that one should always look at the overall ANOVA and then interpret contrasts/post-hoc comparisons. But then again, I generally do post-hoc comparisons only. So it's good to know that contrasts alone are the only comparisons I should look at. Do these retain the same alpha level (.05) or should they be reduced? Also, thanks, I am well aware of the multiple ANOVA problem and the possibility of MANOVA. – Lee May 21 '12 at 23:16
1

Yes. You do need to control the experimentwise alpha when doing planned comparisons. – Joel W. May 29 '12 at 10:23
5

The key nuance here, is whether you have "planned comparisons" (as JoelW. discusses), or "post-hoc comparisons" (like you mention). If your comparisons were decided a-priori (& they aren't just to look at all pairwise contrasts), then JoelW.'s advice is good, but if you gathered some data, looked at it, noticed that some contrasts looked interesting & decided to test those, then you need to exercise more caution: eg, no post-hoc tests w/ non-significant ANOVA, & use alpha adjustments, etc. – gung - Reinstate Monica Jun 08 '12 at 01:47
Thank you for the answer to Lee's question, which parallels mine. Do you happen to know of any published work that would justify looking at planned comparisons only, even in the absence of a significant three-way interaction in ANOVA? Thank you in advance for whatever recommendations. – Sep 08 '13 at 17:20

score 0 · Answer 2 · answered Jun 08 '12 at 01:37

Yes you need to adjust alpha, especially if you do planned comparisons that are non-orthogonal. If you have set the contrasts in an ANOVA and tested the orthogonality between them, you can descompose the SC and df of the ANOVA. The planned comparison are always more powerful than traditional post hoc test, but in cases where the contrast are non.orthogonal you have inflated the experimentwise error. you can check for orthogonality among contrast in many textboxs. However i don´t undestand if you have set that time was factor with two levels, why are you performing a contrast between those times?

What if an overall ANOVA is not significant, but specific contrasts are?

2 Answers2

Linked