I was just asked to familiarize myself with some methods looking at comparing AUROC for a few predictive scores to predict outcomes. Issue is that I have a dataset of about 200 with <5% with the outcome of interest. Given this low event rate is there any issues I should be aware of (or corrections to apply if they exist) for this issue?
Asked
Active
Viewed 25 times
1
-
1Welcome to Cross Validated! It seems that AUROC has the same meaning in the imbalanced setting as in the balanced setting. That you only have ten members of the minority category, however, strikes me as highly problematic. – Dave Oct 25 '23 at 02:58
-
Many pages on this site discuss the sample-size issue for binomial models, which is a big problem with your data. See this answer, this answer, other answers on those pages, and the links from them. To illustrate the problem, do random samples of size 200 with a 5% success rate and see how variable the estimates of a simple success rate can be. Then consider the additional problems introduced when you're trying to evaluate predictors of success. – EdM Oct 25 '23 at 14:46