0

enter image description hereI'm trying to plot the ROC Curve for a number of target/object detection models and compare their performance. The pre-trained models in question take an input image and they output a mask image where each pixel is either white or black. I modified the code of these models such that, right before model decides if each pixel is white or black (1 or 0), I apply a 100 point threshold (values between [0.00, 1.00]) and get 100 set of masks accordingly, then I calculate TP, TN, FP, FN, TPR and FPR with respect to the result of the applied threshold where I set another threshold (like a radius) to tell if the center of each white blob in the mask lies within the circular area of the ground truth coordinates. So, after obtaining 100 sets of TPR and FPR pairs I plot the ROC Curve but the characteristics of the curve seems a little off. I believe there is a logical problem here and I need some advice. All I know is that the TPR peaks then drops a little because as the threshold gets loose the white blobs in the corresponding masks get bigger thus their center gets shifted which results in a false prediction.

Tungdil
  • 1
  • 1
  • What you describe is clearly not a ROC curve, and the plots you show clearly aren't ROC curves either... What's your question exactly? – Calimo Mar 08 '24 at 08:56
  • Isn't ROC Curve used to plot the relation between the TPR and FPR over a changing threshold? If that's the case what's wrong with the result I present? If that's not the case what is the correct way of plotting the ROC Curve for this case? Which steps are wrong and how to logically fix it? Thanks @Calimo – Tungdil Mar 08 '24 at 10:18
  • It is, but not the way you're doing it. See https://stats.stackexchange.com/q/105501/36682 – Calimo Mar 08 '24 at 10:51
  • Hey again, I gave the link you provided a shot and I believe I figured out how it works but I'm unable to map this approach to segmentation-based object detection. But the key points ı believe I figured: -In order to plot a ROC Curve we actually don't need a set of test examples. It would theoretically work since even a single frame contains a number of pixels. So me modifying the code of the models to get a set of examples for each threshold is wrong. – Tungdil Mar 08 '24 at 12:21
  • Some things still confuse me: -I thought ROC Curve was a target level metric but this approach we talk about sounds a lot like a pixel level metric to me since it involves pixel level decisions. -Do you think the calculation method for TP, TN, FP and FN are still applicable or should I change up some things. If so what would you suggest. Thanks a lot @Calimo – Tungdil Mar 08 '24 at 12:25

0 Answers0