0

Should I calculate the classification performances (AUROC, AUPR etc) before or after the neural network calibration (using for ex. isotonic regression)?

andrea_a
  • 13
  • 4
  • It depends on the statistic. ROC AUC is a statistic of ranks so any calibration that does not change the order of the scores also won't change the value of the AUC statistic. On the other hand, other statistics work differently. – Sycorax Jun 03 '20 at 22:26
  • Thank you. If I use isotonic regression, would it change the ROC AUC? – andrea_a Jun 03 '20 at 22:27
  • Does isotonic regression change the order of the data points' scores? That is, is the isotonic regression that you're using non-decreasing or is it non-increasing? – Sycorax Jun 03 '20 at 22:28
  • It is non-decreasing – andrea_a Jun 03 '20 at 22:33
  • Then you have your answer: a non-decreasing function won't re-order its inputs! – Sycorax Jun 03 '20 at 22:35
  • What do you mean by saying that the ROC AUC depends on the order of the scores? – andrea_a Jun 03 '20 at 22:39
  • 1
    I mean that the scores themselves (0.15, 0.63, 0.99) don't matter, but the ordering of them (smallest, middle, largest) does matter for ROC AUC. See: https://stats.stackexchange.com/questions/105501/understanding-roc-curve/105577#105577 – Sycorax Jun 03 '20 at 23:09

0 Answers0