1

So I just learned about AUROC. When I read this thread, it seems like AUROC is not a great metric for imbalanced dataset. One answer even says it shouldn't be used to compare models.

However, I am confused because research papers use AUROC to test models for MIMIC-III code prediction, which is a highly imbalanced dataset. The papers doesn't clearly explain why they picked such metric.

My questions are

  1. Why do you think the authors picked AUROC? Is there an pro for using it that I am unaware of?
  2. Should the SOTA model be MSMN(Highest AUROC) or RAC(Highest F1)? This is confusing because the site is sorted by AUROC on default.
Matt Yoon
  • 135

1 Answers1

2

The AUROC, better understood as the concordance probability between predicted and observed values, has no problem with highly imbalanced data other than having a higher standard error than when the outcomes are balanced and the total sample size stays the same. The problem with AUROC is that like other measures you mentioned it is not sensitive enough for comparing two models. For that one should use the gold standard (log-likelihood) or sensitive measures discussed here.

Frank Harrell
  • 91,879
  • 6
  • 178
  • 397