Questions tagged [false-discovery-rate]

The False Discovery Rate (abbreviated as FDR) is an expected fraction of rejected null hypotheses that are falsely rejected, i.e., the fraction of "significant" findings that are actually not true. These are called "false discoveries". Given $V$ false discoveries and $R$ total rejected hypotheses, FDR can more formally be defined as

$$ FDR = E\left[\frac{V}{R}\right] $$

Controlling the false discovery rate has become a popular method for dealing with the multiple comparisons problem, and has seen wide acceptance in a variety of fields.

Benjamini and Hochberg were the first to introduce this method in 1995 [1]. Their method works as follows:

For a given $\alpha$, find the largest number of tests, $k$, out of all tests conducted, $m$, such that $P_k \leq \frac{k}{m}\alpha$ and subsequently reject all hypothesis $H_i$ for $i = 1 ... k$.

It was later shown by Benjamini and Yekutieli that the above mentioned method is robust to several dependency conditions, and specifically to a subset known as positive regression dependency. They also extended the method to include different kinds of dependency [2].

There have been several modifications and extensions to the FDR method proposed by Benjamini and Hochberg, including notably:

1. The $q$-value extension by John D. Storey implemented in the qvalue R package available on Bioconductor and Github [3,4]. See also this web Shiny implementation of the qvalue R package [10].
2. Local false discovery rates, implemented in the R package fdrtools on CRAN [5,6].
3. Stratified FDR (sFDR) as implemented in Lei Sun's Perl script SFDR [7,8].

The original procedure (sometimes known as the BH procedure) is available as a default method in many software packages, and is an option in the p.adjust(p, method = "BH") function in R. The extended work of Benjamini, Hochberg, and Yekutieli is available through p.adjust(p, method = "BY") [9].

References and Further Reading

[1] Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statisitical Society, Series B, 57(1), 289–300.

[2] Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency, 1165–1188. http://doi.org/10.1214/aos/1013699998

[3] Storey, J. D. (1995). A direct approach to false discovery rates. J. R. Statist.Soc. B.

[4] https://github.com/jdstorey/qvalue

[5] Efron, B., Tibshirani, R., Storey, J. D., and Tusher, V. (2001). Empirical Bayes analysis of a microarray experiment. Journal of the American Statistical Association, 96, 1151-1160.

[6] https://cran.r-project.org/web/packages/fdrtool/fdrtool.pdf

[7] Sun, L., Craiu, R. V., Paterson, A. D., & Bull, S. B. (2006). Stratified false discovery control for large-scale hypothesis testing with application to genome-wide association studies. Genetic Epidemiology, 30(6), 519–530. http://doi.org/10.1002/gepi.20164

[8] http://www.utstat.toronto.edu/sun/Software/SFDR/

[9] https://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html

[10] http://qvalue.princeton.edu/