2

I have a binary classifier which outputs a given score to differentiate normal (low score) from abnormal (high score) cases. The score itself however is non-interpretable to others.

I know a ROC plot is typically used to select a threshold to map these continuous scores onto binary decisions. The threshold is selected based on the desired TPR/FPR tradeoff.

My question is: Can this process be done "in reverse", where instead of selecting a binary threshold and them presenting the end user a decision (normal/abnormal), I present the end user the score and say "cases with this score or higher have the following sensitivity (TPR) and specificity (1 - FPR)"

DankMasterDan
  • 1,486

2 Answers2

1

Your model presumably can provide the probability of being in the "abnormal" class as a function of your "non-interpretable" score. A plot of probability versus score would be more useful to users than forcing them to wrap their heads around sensitivity and specificity estimates.

If a user is really forced to make a class assignment based on this single score, then the probability cutoff is based on the relative costs of false-positive and false-negative assignments. If there are other types of information available, the probability based on this score will be most useful to combine with them.

Frank Harrell discusses problems with specificity and sensitivity, and the advantages of direct probability estimates, here.

EdM
  • 92,183
  • 10
  • 92
  • 267
  • (1) the model itself provides a score (not bounded to [0,1]). I am not sure of any clear-cut way to go from this score to a "probability of being in the "abnormal" class" (2) this is in medical context so specificity/selectivity very standard to users. – DankMasterDan Jul 24 '22 at 06:02
  • @DankMasterDan if all you have is accuracy/specificity/sensitivity for a classifier, the model runs several risks, particularly in the medical context. See the Harrell link in the answer, this Cross Validated page, and this post. – EdM Jul 24 '22 at 12:10
  • @DankMasterDan this post shows the dangers of relying on ROC in the medical context. – EdM Jul 24 '22 at 12:20
0

If you only have the scores and no ground truth targets, and you would assign targets according to a chosen threshold for those scores and use those labels as ground truth labels, then you would always get a perfect ROC score equal to one. For the ROC curve to make sense, you need ground truth targets.

The standard situation is that you are given ground truth labels, that you have learned a binary classifier from those ground truth labels via supervised learning, and that this classifier provides you with a score. Then the ROC score is independent of the threshold you choose for predicting the label of future observations. The ROC curve only depends on the ordering given by the score and on the ground truth variables.

Now to your question: Yes, you can create a binary classifier with any threshold and then report the sensitivity and specificity thereof. This is often done and helps users to choose the classifier they prefer (some might favor high recall, others are more afraid of false positives).

frank
  • 10,797
  • to be clear I am doing the latter: I train a classifier with ground truth labels. Then I apply that classifier on a sample for which I dont have ground truth and get an output score. Based on the previously obtained ROC, I would like to make a statement about the specificity/sensitivity of declaring this sample positive. – DankMasterDan Jul 24 '22 at 06:09