3

I am aware of that multiple testing correction is needed when multiple hypothesis tests are simultaneously performed. However, I am a little confused by the word "simultaneously".

Say I have hundereds of samples and I derived multiple features (variables) from each sample. With this sample-feature matrix, I would like to perform pairwise fisher exact test for feature co-occurrence. Do I need to carry out multiple testing correction on the P-value of each paired feature?

I guess multiple testing correction is needed. However, I can consider that each of the pairwise fisher exact test was carried out individually and each raw p-value represents the significance of correponding paired features, right?

unicorn
  • 889

1 Answers1

1

If you are going to look at lots of relationships separately and then claim you have discovered something when $p\leq 0.05$, then this is a multiplicity issue (i.e. if you only do enough tests you will appear to "discover" something). This has almost nothing to do with what your analysis method is (except that some hierarchical models might be said to have some multiplicity protection "build-in").

In fact, with hundreds of comparisons made, you will need to do something to be taken seriously with any claim based on $p\leq 0.05$. Whether the something is to control the familywise type I error rate, or say control the false discovery rate, or something else is to some extent up to you (but straying too far from the conventions of your field may not be a good idea).

Björn
  • 32,022
  • Here is what I am confused about: Say we have a pair of variables that gives insignificant p-value only after correction is applied. If I happen to calculate this one paired variables alone, it will still be significant because correction is not needed in such case, right? It is seems contradictory to me. We got different result for this very variables pair although nothing changed. – unicorn Apr 18 '19 at 13:50
  • The difference in the frequentist null hypothesis testing framework is that it's not the same thing whether you have a single pre-specified hypothesis that you wish to confirm vs. going on a fishing expedition looking for statistical significance amongst hundreds of possible associations. I.e. the difference is whether you predicted it before your experiment and are confirming it or not. To be honest with the number of things you are looking at, it sounds more like you are in potential hypotheses generating mode. – Björn Apr 18 '19 at 19:52