4

Suppose, I have three time-points, Ta, Tb and Tc. Let Ta be control and Tb and Tc be the effect of a drug 4 hours and 8 hours after treatment. For each pair of time-points, I compare about 15000 observations (genes, for differential expression, to be precise). So, for each pair of time-point, for each gene, I obtain a p-value.

My objective is to obtain ALL those genes that might have had an effect due to the drug treatment. Now, in order to correct for multiple testing, I use BH method. However, I have a confusion regarding the way to apply FDR.

Case A: I could pool ALL the p-values from ALL pairs of time-points together (Ta.Tb, Tb.Tc, Tc.Ta, each gene occuring thrice) and correct for multiple-testing once on this pooled set.

Case B: I could correct for multiple testing within each pair of time-point (separately for Ta.Tb, Tb.Tc, Tc.Ta). I am not sure if there is a consensus as to which one to employ and why.

My understanding: For case A, suppose there are too many "highly" significant events between Ta and Tb, then, you "might" lose the events that are otherwise statistically significant (with p-values computed), i.e., they become insignificant after FDR correction due to low p-values in other time-point pairs. For example, if the overall effect of drug (meaning for most of the genes) is NOT as strong in Tb.Tc compared to Ta.Tb, then we might not see that there is an effect of the drug at Tb.Tc at all.

And in case B, you would probably get more significant events, meaning more false positives as well?

I'd greatly appreciate it if someone could clarify this.

Thank you very much.

Arun
  • 844

1 Answers1

1

Your case A is a better approach because your Case B does not attempt to control the overall false discovery rate. (That is, if you set q to be 0.05 for each pair of time points then your overall false discovery rate is going to be bigger than q).

However, a problem with your Case A is that if applying BH's original FDR method it assumes independence between the tests which is clearly not true in the case of your experiment (although the FDR can perform surprisingly well when the independence assumption is not met). One way of attempting to resolve this is to test for a independence between the three-level time factor and the effect of the drug (i.e., rather than conducting the testing pairwise).

Another approach is to jointly test all of the differences between each pair (e.g., using MANOVA) and then discard pairs of time periods that are not jointly significant prior to applying the FDR.

Also, parenthetically, there is a Case C which you have not considered which is the default in lots of analysis of surveys and it involves controlling the familywise error rate within the pairs and ignoring it for the genes. (I mention this only for completeness and am not suggesting it makes sense in your experiment.)

Tim
  • 3,401
  • Thanks Tim for your reply. What I'd like to know is if both cases are in practice amidst the trade-offs. If I got it right, my observations are identical to yours for cases A and B; the only difference being that I prefer case B and choose to live with the "multiplicity problem". I checked about FWER, but it might be too conservative - either Holm's or Bonferroni's method (no restrictions on joint distributions as opposed to Hochberg's). I don't see a difference between the "normal" BH method and the FWER Hochberg's step-up procedure. Please correct me if I am wrong. – Arun Nov 21 '12 at 08:16
  • The answer given here by JohnRos is more or less the point I try to make: It'd be nice to get your take on it. http://stats.stackexchange.com/questions/26588/multiple-fdr-corrected-experiments-using-the-same-data?rq=1 – Arun Nov 21 '12 at 08:39
  • You are warning to use BH’s procedure to control the FDR for the experiment and B won’t do this. It will also give you practical problems. What will you do if you reject the null for a gene in one pair but accept in another pair even though it has a lower p-value? Having said that, Case C, which is the norm in survey research, suffers from the same problem but remains popular; I am hoping that somebody smarter than me will post a better answer! (By FWER I am referring to the family wise error rate which motivated multiple comparison corrections pre-BH.) – Tim Nov 21 '12 at 23:48
  • I agree, except the part about practical problems. Do you think rejecting null for a gene in one-pair but accepting in another would not happen in case A? I have 3 pairs of time points and so each gene will be occurring thrice in the pooled-dataset. It might as well be that gene from 1 pair alone passed FDR cut-off, isn't it? At the moment, I consider genes that are significant in at least 1 pair. – Arun Nov 22 '12 at 10:14
  • 1
    If your approach is to treat the "significance" of a single pair as enough to conclude that the gene is significant as a whole then I think you do need to be jointly testing all three pairs (e.g., using an ANOVA if your data is appropriate). To test each separately and controlled under separate FDR seems to me to be an unambiguous violation of what you are wanting to achieve by controlling the FDR. – Tim Nov 27 '12 at 04:50