Denominator is Zero for Matthews correlation coefficient and F-measure

Question

Recently, I built a classification model based on the imbalanced data set(positive sample is minority and negative sample is majority), and the model gave the following result for the test set:

True Positives = 0

True Negatives = 139

False Positives = 0

False Negatives = 10.

My question is: for the result, can Matthews correlation coefficient (MCC ) and F-measure be used for estimating the classifier?

Since the denominators for MCC and F-measure are zero, it seems meaningless. If so, MCC and F-measure is not always works for estimating the classifier, and sensitivity and specificity as well as g-mean should be better. Is that right?

Any help is appreciated.

Answer 1

This is only really a problem if you compute the precision and recall first, then plug them in.

One can also compute the $F_1$ score as $$F_1 = \frac{2 \cdot \textrm{True Positive}}{2 \cdot \textrm{True Positive} + \textrm{False Positive} + \textrm{False Negative}}$$

Plugging in your numbers, you'll arrive at an $F_1$score of zero, which seems appropriate since your classifier is just guessing the majority class.

There is an information-theoretic measure called proficiency that might be of interest if you are working on fairly unbalanced data sets. The idea is that you want it to remain sensitive to both classes as either the number of true positives or negatives approaches zero. It's essentially $$ \frac{I(\textrm{predicted labels}; \textrm{actual labels})}{H(\textrm{actual labels)}}$$

See pages 5--7 of White et al. (2004) for more details about its calculation and interpretation

Answer 2

I wanted to provide a more complete answer to address both F and MCC, and to provide a well reasoned alternative definition to avoid undefined-ness in both cases.

When $F_\beta$ is undefined

In one sense, the F-measure is truly undefined when precision or recall are 0, which happens when there are zero true positives:

F-measure (originally introduced in MUC-4, 1992): $$ F_\beta = \frac{(\beta^2 + 1.0) \times P \times R}{\beta^2 \times P + R} $$ where $P = \frac{TP}{TP + FP}$ and $R = \frac{TP}{TP + FN}$. Here, when $TP = 0$, precision and recall are 0, and $F_\beta = \frac{0}{0}$, which is undefined.

However, as far as $F_1$, when precision and recall are given equal weight ($\beta = 1$), you can rewrite without computing precision and recall, and avoid some undefined bounds: $$ F_1 = \frac{2 \times TP}{2 \times TP + FP + FN} $$ In this case, $F_1 = 0$ when $TP = 0$ AND $(FP \neq 0$ OR $FN \neq 0)$.

BUT, $F_1$ is still undefined when TP, FP, and FN are all 0!

Handling undefined-ness

Some say those are cases when $F_1$ is an unnecessary test. Perhaps so, if you're just running it once. I have found the following alternative useful when computing many $F_1$ scores on models that may include undefined cases, and I want to compare them all fairly.

Thanks to https://stats.stackexchange.com/a/305905/240083, I found https://github.com/dice-group/gerbil/wiki/Precision,-Recall-and-F1-measure, which provides a nice solution that avoids the divide by 0 issue when TP, FP, and FN are all 0:

In some rare cases, the calculation of Precision or Recall can cause a division by 0. Regarding the precision, this can happen if there are no results inside the answer of an annotator and, thus, the true as well as the false positives are 0. For these special cases, we have defined that if the true positives, false positives and false negatives are all 0, the precision, recall and F1-measure are 1. This might occur in cases in which the gold standard contains a document without any annotations and the annotator (correctly) returns no annotations. If true positives are 0 and one of the two other counters is larger than 0, the precision, recall and F1-measure are 0.

I think this applied well to any binary classification problem. For example, if the predicted classes are all 0, and the true classes are indeed all 0, then $F$ should capture a perfect classification.

MCC's undefined-ness

This idea extends well to MCC too, albeit more elaborately. MCC ranges from $[-1, 1]$, where 1 is perfect classification, 0 is arbitrarily guessing, and -1 is a perfectly incorrect classification.

$$ MCC = \frac{(TP \times TN - FP \times FN)}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}} $$

MCC is undefined in four cases:

1. if $TP = 0$ AND $FP = 0$, then $MCC = \frac{0}{0}$ = undefined
2. if $TP = 0$ AND $FN = 0$, then $MCC = \frac{0}{0}$ = undefined
3. if $TN = 0$ AND $FP = 0$, then $MCC = \frac{0}{0}$ = undefined
4. if $TN = 0$ AND $FN = 0$, then $MCC = \frac{0}{0}$ = undefined

A proposed alternative; for each of the four cases above, there are three valid subcases to consider (when MCC = 1, 0, or -1):

1. if $TP = 0$ AND $FP = 0$, then the classifier predicted all negatives, and we need to check $FN$ and $TN$:
   1. if $FN = 0$ AND $TN \neq 0$, then the predicted and true classes are all negative, and the classifier is perfectly correct, so $MCC = 1$.
   2. if $FN \neq 0$ AND $TN \neq 0$, then the true classes are mixed and the classifier is arbitrarily wrong, so $MCC = 0$.
   3. if $FN \neq 0$ AND $TN = 0$, then the true classes are all positive, so the classifier is perfectly wrong and $MCC = -1$.
2. if $TP = 0$ AND $FN = 0$, then the true classes are all negative, so we need to check both $FP$ and $TN$:
   1. if $FP = 0$ AND $TN \neq 0$, then the classifier is guessing all negatives, so the classifier is perfectly correct and $MCC = 1$.
   2. if $FP \neq 0$ AND $TN \neq 0$, then the guesses are mixed and the classifier is arbitrarily wrong, so $MCC = 0$.
   3. if $FP \neq 0$ AND $TN = 0$, then the classifier is always guessing positive, so the classifier is always wrong, so $MCC = -1$.
3. if $TN = 0$ AND $FP = 0$, then the true classes are all positive and we need to check $TP$ and $FN$:
   1. if $TP \neq 0$ AND $FN = 0$, then the classifier is guessing all positives, so $MCC = 1$.
   2. if $TP \neq 0$ AND $FN \neq 0$, then the guesses are mixed and the classifier is arbitrarily guessing and $MCC = 0$.
   3. if $TP = 0$ AND $FN \neq 0$, then the classifier is only guessing negatives and it is perfectly wrong, so $MCC = -1$.
4. if $TN = 0$ AND $FN = 0$, then the classifier is always guessing positives, so we need to check $TP$ and $FP$:
   1. if $TP \neq 0$ AND $FP = 0$, then the true classes are also all positive and $MCC = 1$
   2. if $TP \neq 0$ AND $FP \neq 0$, then the true classes are mixed and the classifier is arbitrarily wrong, so $MCC = 0$.
   3. if $TP = 0$ AND $FP \neq 0$, then the true classes are all negative and the classifier is perfectly wrong, so $MCC = -1$.