5

I have learned many models and I calculated p-values for the cross-validation errors. I want to select significant models based on the false discovery rate (FDR). How can I estimate the FDR from p-value distribution?

kjetil b halvorsen
  • 77,844
user2806363
  • 2,723
  • I strongly recommend looking at Andrew Gelman’s writings on this. Most statisticians would argue that the true false discovery rate is always 0%, because the null hypothesis is always false. – Closed Limelike Curves Mar 20 '22 at 00:49
  • Why are you trying to estimate the false discovery rate? What’s your goal in your study? Are you trying to adjust for multiple comparisons? – Closed Limelike Curves Mar 20 '22 at 01:01

1 Answers1

-1

Generally speaking, the false discovery rate correction requires ranking your p-values and then computing a critical value (based on the false discovery rate) that you can compare to your p-values. This website gives a very concrete example: here.

If you're using R, then it's even easier to do the adjustment. The following code is a quick example:

pvals <- c(0.04, 0.03, 0.06, 0.01, 0.02, 0.003)
p.adjust(p = pvals, method = "BH")

In the above code, the object pvals is a list of the p-values that you obtained from your analyses. The function p.adjust provides several different options for adjusting p-values based on multiple observations. In this case, the false discovery rate adjustment can be done by specifying method = "BH" for Benjamin-Hochberg or method = "fdr" for false discovery rate (note: they are the exact same procedure and will give the exact same results).

Billy
  • 826