1

When attempting to construct a classifier for a somewhat imbalanced data set, I was led to the question of measuring the performance of the classifier. One of the first things I thought of was to take the average of precision and specificity:

$$\frac{(\text{true positive rate} + \text{true negative rate)}}{2}$$

It seems quite an intuitive measure to me. Moreover one can take a weighted average depending on how important the prediction rate for each class is. Also it generalizes in a straightforward way to multiple classes. One can incorporate it into a 2-class cost matrix by setting the off diagonals to:

$$\frac{(N-N_1)}{N}, \frac{(N-N_2)}{N}$$

However, I could not find any reference online to any such measure - there appears to be no name for it! There are all sorts of combinations of other measures, but not this one. I was wondering if someone can point me to any discussion of it.

Edit: I originally had misunderstood the definition of specificity as $TN \over TN+FN$ when actually it is defined as $TN \over TN+FP $. I have corrected the title. As far as I can find, there is no name for "true-negative rate".

Djib2011
  • 5,933
  • 5
  • 28
  • 38
skm
  • 113
  • TN/(TN+FN) is the Negative Predictive Value (~"negative precision"). TN/(TN+FP) is specificity. And it is "true negative rate" (or ~"negative recall"). – Narfanar Jan 29 '20 at 15:32

1 Answers1

2

What you are proposing is the (weighted or not) arithmetic mean of precision and specificity.

Two of the most common (confusion matrix) metrics being used are:

  • F1-score. This is essentially the harmonic mean of precision and recall.
  • Fβ-score. A weighted version of the above. It measures the effectiveness of retrieval with respect to a user who attaches β times as much importance to recall as precision.
  • Geometric Mean Score. It corresponds to the geometric mean of sensitivity and specificity and is used mostly in imbalanced problems.

Your question essentially comes down to:

Why are the harmonic and geometric means preferred to the arithmetic mean for averaging confusion matrix metrics?

An excellent answer concerning these three means and their diffenences can be found here. Also, a discussion on which mean to use and when can be found here.

Djib2011
  • 5,933
  • 5
  • 28
  • 38
  • Thanks for the information.

    According to the answer at this link the average of the per class accuracy is not a "proper score function" - but I'm not convinced. I have to read up on some of the classification terminology and get back.

     – skm Apr 24 '18 at 09:23
  • 2
    Nothing mentioned above is a proper accuracy score. And are you certain it was appropriate to cast this problem as a classification problem and not one of risk estimation? Describe the problem more fully. And note that imbalance is to be expected, so it's not clear why you mentioned that. – Frank Harrell Apr 24 '18 at 12:16