2

I understand that applying Bonferroni correction involves dividing alpha by number of tests performed. However, I am unsure about how I can determine the number of tests to divide by.

I have 6 IVs that are entered into a simultaneous equation to predict a DV. For exploratory analysis, the DV is further split into 3 sub-scores, each measured across 4 time points. The 6 IVs are then measured for each score at each time point. Does this means that I would need to divide alpha by 12 (3 scores * 4 time points), 6 (6 IVs) or 72?

Thanks in advance!

hrhy
  • 35

1 Answers1

2

Like for many things in statistics there is no fixed rule about how the Bonferroni correction is to be applied. The Bonferroni correction is meant to achieve a certain aim. If you have $k$ tests and a level of $\alpha$, and you apply the Bonferroni correction over these $k$ tests, it means that assuming all $k$ null hypotheses to be true, the probability of finding at least one significant rejection in your $k$ tests is smaller or equal than $\alpha$.

So in your situation, if you divide by 72, the probability to find any wrong significance is smaller or equal than $\alpha$. This is the safest thing you can do, but it has the obvious disadvantage that also the power, i.e., the probability to find a "correct" significance in case that a null hypothesis is false, is lower than if you use a lower $k$ for the Bonferroni correction. Some people would divide by 6 for the IVs (so that their probability for finding a wrong significance among those tests only is $\le\alpha$), and then by 12 for each of the sub-score/time point-tests, to secure the $\alpha$ for each group of them in isolation, which has a better power than dividing by 72, but less protection against wrong significances.

The approach that I would probably take is to report results at various levels, like "tests A and B are significant at level $\alpha/72$, which secures an overall level $\alpha$ by means of Bonferroni. A number of further tests still give an indication that something may go on, which would deserve further investigation. Test C and D are significant at $\alpha/6$ (Bonferroni applied to IVs only), and test E and F are just significant at level $\alpha$, meaning that these could easily be meaningless, however they give some weak indication against their null hypotheses."

  • Hi Lewian, thank you so much for your explanation! Could I clarify if I would need to apply the corrected alpha to the regression model (F-stats) as well? – hrhy Oct 03 '20 at 16:40
  • As I wrote, there is no rule. The theory tells you the implications and you have to decide which implications you want to have. Most people wouldn't include the F-test in Bonferroni counting, but then they often wouldn't run any other test in case the F-test is insignificant. – Christian Hennig Oct 03 '20 at 17:20