Does ROC assessment of a binomial classifier serve as a good performance measure given equal weights between positives and negatives?

Question

My problem is that i have created four candidate models that I am comparing mainly via the following performance measures: F-measure, recall, precision, accuracy and visual ROC assessment.

The problem is that as you see from the table, the SVM_linear_test performs the best given F-measure. This model corresponds to the blue line in the ROC chart. The red line in the ROC charts corresponds to the SVM_RBF_test as given the ROC chart this model performs the best. Having read a lot recently about performance measures on binomial classifiers I have not come across what is obvious in my example here. ROC does not take into account false positives and therefore in my case, a ROC assessment is not worth much. One can of course always debate if we would want to assign more weight to the positive class but in this case we leave this discussion out.

Numerical performance measures

enter image description here

score 2 · Answer 1 · answered Mar 26 '13 at 16:57

2

We know from statistical theory that in the absence of prior information, the log likelihood is an optimum criteria for estimating unknown parameters such as regression coefficients. Because of that it is very advantageous to use the log likelihood in judging model performance. This leads to optimum power and precision. So think about using a log-likelihood-based generalized $R^2$ measure. This solves the problem that ROC area is too insensitive to detect real differences in predictive discrimination. In addition, the ROC curves you have drawn, while often used in standard practice, have an enormous ink:information ratio. ROC curves are usually drawn because of their frequency of use in the literature, but I have not seen an example where looking at the curves leads to insights.

answered Mar 26 '13 at 16:57

Frank Harrell

91,879
6
178
397

Thanks for the advice Frank! The thing is that only very rare are log likelihood available within machine learning, which is the area where one applies the ROC curve, or at least where i apply it. – Kasper Christensen Mar 26 '13 at 17:26
1

Then consider the fully proper scoring rule known as the Brier score (quadratic error). – Frank Harrell Mar 26 '13 at 18:03
I never heard about that measure. I am using it for machine learning... – Kasper Christensen Mar 28 '13 at 23:45
1

Brier score (used since 1951 for judging accuracy of rainfall forecasts) is the mean squared difference between predicted probability and 0/1 outcome. – Frank Harrell Mar 29 '13 at 12:51

Does ROC assessment of a binomial classifier serve as a good performance measure given equal weights between positives and negatives?

1 Answers1