3

I'm trying to better understand the ROC when used for machine learning model classification and was looking at this explaining curve, explaining what is better and worse. However, I am thinking, contrary what is shown on the curve, that perhaps worse is really whatever is closer to the random distribution. Because whatever is not random can always be functionally inverted. I.e. A curve well under the diagonal would actually be better than the diagonal?

Is an off (below) diagonal ROC curve not always better than random?
(Please, note that none of the previous answers actually address this question.)

enter image description here

Related topics:

Dave
  • 62,186
not2qubit
  • 135
  • 3
    You want to take the opposite labels of a bad classifier...understandable. Let's draw an analogy. I predict if the weather will be rainy or sunny, and I'm inaccurate. I often predict sunny days as rainy and rainy days as sunny. You catch this, flip my predictions, and get good results. Am I suddenly good at predicting the weather? (Further, the model you get by flipping the categories of a poor classifier still suffers from issues related to probability predictions.) – Dave Oct 25 '20 at 18:12
  • Thanks Dave, but can you expand on the statement "...flipping the categories of a poor classifier still suffers from issues related to probability predictions.", in non-stat language? – not2qubit Oct 25 '20 at 19:13

1 Answers1

1

It might be that you can calibrate the predictions in order to reverse the terrible model predictions to be good, as is suggested in the comments here. However, you need to know to do this, and I dislike that some popular software packages automatically flip the ROC curve when it falls below the diagonal. This puts the analyst in a position to use predictions that might be terrible while thinking those predictions are good. No! Those predictions are terrible. They might be able to be transformed into good predictions (calibration), but the predictions themselves are terrible.

Separate from pure predictions, if the model makes terrible predictions that can be made good through some calibration, interpreting the coefficients of the original model, as is common to do in many areas of science, is at least a bit dubious. If you calibrate the predictions and consider the final predictions to be the original model and then the calibration, then the coefficients from the original model no longer apply to the two-step strategy.

Dave
  • 62,186