2

I have two independent variables, both of them only have 2 levels (Story type: Homosexual vs. Heterosexual, and Stereotype: High vs. Low).

I run three two-way ANOVAS, to test the IVs' effects on three continuous dependent variables.

Now I need to do a Bonferroni Correction for the multiple comparison. I am confused that I should use 0.05/3, because I run the two-way ANOVA three times. Or I should use 0.05/6, because there are six p-values in the study (no significant interaction has been shown).

Yanyan Zhou
  • 23

1 Answers1

0

Corrections are made for family-wise error rates, so you effectively need to correct for each p-value that you calculate that you believe to be in the same family of hypotheses. It is hard to tell you exactly whether this needs to be three or six, but my guess would be six as it sounds like all three tests are related. Some of the answers here might illuminate this further and help you decide.

Although, this is something of an aside, you can avoid some of the multiple comparison issue by conducting a multivariate analysis of variance (MANOVA) by effectively combining all three of your response variables into one and simultaneously testing if these differ between your two covariates, meaning you only need to correct for two comparisons instead of three or six.

André.B
  • 1,450
  • 1
    Thank you so much for your response. I am still a little bit confused. I in fact run a Two-way MANOVA first, and both of the two independent variables are significant on all the three dependent variables. Do I still need to do a post hoc using the ANOVA and do I need to adjust the p-value? If I correct the Two-way MANOVA, I only need to correct for two comparison instead of three or six? – Yanyan Zhou Apr 17 '19 at 15:00
  • MANOVAs usually output a single p-value for each covariate (ind. variable) you include. Since you have two covariates then it should have one p-value for each, and you could correct for each of these. What software are you using to do this? If you post a sample dataset I can probably help you more. – André.B Apr 17 '19 at 20:53
  • The real difference is what a MANOVA is capable of telling you v.s. an ANOVA. A MANOVA can tell you if there is a statistically significant difference between at least one of the levels of your covariate and the others on at least one of your response variables. However, it does not tell you which level differs from which others and how, or on which response variable they differ, just that they differ on at least one. – André.B Apr 17 '19 at 20:58
  • In your case, since each covariate only has two levels then a signficant p-value for that term tells you that say homosexuals are different from heterosexuals on at least one of the response scales. Same for stereotype. You may want to take this a step further and include an interaction term between your covariates though, so you can see if the level of one affects the response of the other (and vice versa). – André.B Apr 17 '19 at 20:59
  • In any case, when producing lots of p-values, be they in multiple ANOVA's or a single MANOVA, it is a good idea to correct for multiple p-values when they sit in the same hypothesis framework. Bonferroni's correction is quite conservative (i.e. it is quite harsh) so you might want to look at things like Tukey's multiple comparison procedure or False discovery rate corrections - these can be a little more lenient and Tukey's is a little more informative. – André.B Apr 17 '19 at 21:02
  • Thank you again for the response. So I use SPSS to run the MANOVA. In the SPSS, except showing the p-value for each independent variable, it also gives me a chart of "Tests of between subject effects". I have added that chart to the original question and I wonder if I still should do the p-value correction with the p-values in this chart. – Yanyan Zhou Apr 19 '19 at 20:18
  • Yes, definitely correct as it looks like you are getting a p-value calculated for each covariate term for each of the response variables. You can probably drop the interaction between the two covariates though, as it is not significant. I hope this answers your question, but if it does I would greatly appreciate an up-tick and having my answer accepted :) Let me know how you get on though – André.B Apr 22 '19 at 21:48