3

I'm looking to run a bunch of t-tests, and I'm trying to figure out the appropriate time to apply an FDR correction.

I have four conditions and am doing pairwise comparisons amongst these conditions, so I have six pairwise comparisons/t-tests to run on each variable. I have 30 variables.

Does it make more sense to:

  • Do all of the pairwise t-tests, take all of the p-values, and perform one FDR correction? I.e.: I have 6 p-values (6 pairwise comparisons) for the 30 variables and run one FDR correction on the 180 p-values?

  • Or to perform an FDR correction for each pairwise t-test for each variable? I.e.: Run an FDR correction for the 6 p-values for each variable, for a total of 30 FDR corrections.

ETA:

To clarify, my variables are not dependent (my title was edited)--perhaps 'variables' is not the correct term. I am looking at changes in gene expression for 30 different unrelated genes. The conditions are the same for each gene though. So for gene 1, I am running pairwise t-tests for condition A vs B, A vs C, A vs D, B vs C, B vs D, and C vs D, and then repeating this for each gene.

I've just noticed this question, in which the asker has a similar set-up to me: Multiple testing and FDR on multiple-pairs

I hadn't thought about running FDR for each pairwise comparison (6 FDR corrections for my case), though I'm not sure this would be the best option. Advice?

Community
  • 1
hmg
  • 131

1 Answers1

1

Benjamini-Hochberg procedure (FDR) generally assumes tests are independent from each other (but see this), so it may not be an ideal test for among-conditions assessment (the 6 pairwise comparisons). Otherwise, 'to run a bunch of tests' sounds like a single-family/question approach so you'd need to apply correction to the entire population of p-values generated. In other words, you just set your bet 180 times, chances you 'win' in this roulette are quite a bit higher than if you were to set 1 bet. A more cohesive design e.g. a two-way ANOVA with Tukey-Kramer post-hoc comparisons, assuming the 4 conditions are the same throughout, would be much preferred.

katya
  • 2,142
  • BH does allow for certain types of dependence! – Christoph Hanck Jul 30 '15 at 19:11
  • 1
    @ChristophHanck - ok, clarified – katya Jul 30 '15 at 19:31
  • To clarify, my variables are not dependent (my title was edited)--perhaps 'variables' is not the correct term. I am looking at changes in gene expression for 30 different unrelated genes. The conditions are the same for each gene though. So for gene 1, I am running pairwise t-tests for condition A vs B, A vs C, A vs D, B vs C, B vs D, and C vs D, and then repeating this for each gene. – hmg Jul 31 '15 at 14:30
  • Note also the paper: Joseph P. Romano · Azeem M. Shaikh · Michael Wolf: "Control of the false discovery rate under dependence using the bootstrap and subsampling." Test (2008) 17: 417–442. The simulation results that are reported there suggest that BH is effective under quite general correlation structures. Not a proof, but BH seems clearly hard to break – John Maindonald Oct 01 '17 at 23:00