2

I would like to ask a question on how to interpret the results of two different models, based on AUROC and F1 metrics. As all of you know, AUROC calculates the area under the ROC curve, and the F1 score is the harmonic mean of recall and precision.

While both of them are used for classification metrics, I wonder how should I interpret the below 2 model prediction performance.

model 1: AUROC: 72.28, F1: 60.89

model 2: AUROC: 87.44, F1: 46.11

My question is, just looking at the above model results, is it possible to compare them? If yes, what is the best explanation for them?

Batuhan B
  • 663
  • 2
  • 7
  • 13

2 Answers2

1

It does not make sense to me to compare these, as they evaluate different data.

The AUC evaluates the outputs of your model.

“But so does the $F_1$ score,” you protest? False. The data used to calculate the $F_1$ score are the model outputs after an additional function has transformed them to hard classifications. This is done by applying a threshold and classifying as positive above that threshold and negative below. Consequently, if your $F_1$ is poor, is might be that your model outputs are trash, or it might be that they are fine and this threshold function is not a good one. Maybe you need a higher or lower threshold than the software default.

My suspicion is that you can find a threshold for the second model that gives a higher $F_1$, since the ROCAUC indicates better ability to distinguish between the categories.

Note that all of this threshold business is of debatable value. The direct outputs of models contain rich information that you destroy when you apply a threshold. For instance, there might be more worthwhile decisions than categories, and we have an academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity, all of which are threshold-based statistics. Thresholds have their use but do not necessarily need to be part of your modeling.

Dave
  • 62,186
  • 1
    "this threshold business is of debatable value." whether the threshold has value depends on the nature of the application. In some applications you do have to make a hard classification and the performance metric that is most relevant to the end application will depend on the threshold. When choosing a metric, we need to consider the needs of the application, rather than impose our favoured metric on the application. – Dikran Marsupial Mar 31 '23 at 22:12
  • +1 to both Dave and Dikran. I think using a cut-off of 0.50 everywhere can often not be an optimal strategy. That said, making the metric relevant to the classification is absolutely relevant and sometimes we have to make "hard classifications". – usεr11852 Mar 31 '23 at 23:40
0

Probably ur class distribution is skewed. Confusion matrix would be more helpful to make a comment about the performance. F1 score and PR curve are popular measure to evaluate models in imbalanced classification. I would prefer the Model 1 since it has higher F1 score. ROC-AUC gives misleading results in the presence class imbalance.

ForestGump
  • 327