2

Suppose we are testing a large number ($p$) hypothesis tests. Most commonly, a correction is applied to control the false discovery rate (FDR) at a low value, say $\alpha$. But now imagine a want to report the top 10 features with the largest test statistics, regardless of whether the corresponding p-values are significant or not. Is there a way to estimate the corresponding false discovery proportion (FDP)? Hence I fix the number of discoveries and want to estimate the FDP, rather than fixing the FDR and letting the number of discoveries vary as it is usually done.

I imagine just taking the 10th smallest Benjamini-Hochberg adjusted p-value or tail area false discovery rate will not be a valid estimator. Any pointers to existing literature or terminology would be appreciated.

Knarpie
  • 1,648

1 Answers1

4

False discovery rates give one false confidence, and ignores the often huge false non-discovery rates. In other words, there may be little confidence that important features weren't missed. I think that a more honest analysis that fully exposes the difficulty of the task is to use the bootstrap to compute confidence intervals on the importance rankings of all candidate features. This provides among other things (1) which "winning" features have least favorable confidence limits that rule out (with 0.95 "confidence") it being a "loser", and (2) which "losing" features have most favorable confidence limits that rule out it being an actual "winner". Examples are here and here, the latter result shown below. Bootstrap confidence intervals of ranks of predictors

Going down the false discovery rate path results in findings that have a very low probability of being correct. The bootstrap approach on the other hand exposes which candidate features are currently unable to be classified as important or not due to an inadequate sample size. Importance can be measured any way you want, all the way from ordinary correlation coefficients squared to odds ratios adjusted for all the other features.

Frank Harrell
  • 91,879
  • 6
  • 178
  • 397
  • Thanks for the answer, but how can the bootstrap help me to estimate the FDR if I always call the top 5 point estimates in your example? I am not worried about missing true winners, since I have resources for following up on precious few findings anyway. I just want to know how reliable this set of top features is. – Knarpie Jul 31 '23 at 12:50
  • The bootstrap directly answers your question about reliability of apparent winners and of apparent loser features. As I tried to describe, FDR does not answer the underlying question. – Frank Harrell Jul 31 '23 at 13:08
  • What is then the expected proportion of false findings based on the bootstrap if I consider the 5 top features in the graph you show? False findings may either be features for which the null hypothesis holds, or just features whose true values are not among the top 5, an answer in either case would be helpful. That is the question I want to answer, irrespective of whether it the 'appropriate' question or not. – Knarpie Jul 31 '23 at 14:03
  • The bootstrap tells you more than that. It tells you whether there is reliability in your declared 'winners' and in your non-winners. – Frank Harrell Jul 31 '23 at 14:07