How to calculate F-Measure from Precision Recall Curve

Question

I have a precision recall curve for two separate algorithms. If I want to calculate the F-Measure I have to use the precision and recall values at a particular point on each curve.

How is this point decided? For example on curve one there is a point where recall is 0.9 and precision is 0.87 and the other curve there is a point of recall at 0.95 and precision at 0.84.

Alternatively, should I plot a F-measure curve for every precision recall value?

Answer 1

It is hard to read off F1 (or any other weighted F-measure) directly from a Precision-Recall graph, because of needing to work with reciprocals (harmonic mean).

But if instead you plot the reciprocal Precision & Recall, then values of the F-measure form isobars (straight lines with equal values) with gradient depending on the tradeoff parameter. In the case of F1 they will be isobars parallel to the diagonal, corresponding to equal weighting of success in terms of precision of the positive predictions and in terms of recall of the positive cases (the number predicted may not correspond to the real number of positives).

If you use isobars directly on the Precision-Recall graph, you optimize the arithmetic mean instead - the advantage of using harmonic over arithmetic is debatable and is discussed in the reference, and there is some evidence that geometric is better than either.

If you plot Precision & Recall logarithmically, then the isobars can be used to optimize this geometric mean.

Note that the logarithmic and reciprocal PR graphs are not defined at the points corresponding to no positives are present or none are predicted. At all other points these curves are equivalent to the ROC curve in that if a solution is better ranked in one it will be better ranked in the other.

ROC curves are however a bit different as the compare TPR (Recall) against FPR (Fallout) but again isobars are useful and those parallel to the diagonal correspond to equal weighting of positives and negatives. In particular Precision and Recall are independent of the number of True Negatives (correctly predicted negatives) but this is a complmenetary component of Fallout - in fact a mirror image graph is formed by plotting TPR vs TNR. The difference TPR-FPR is also known as Youden J or Informedness, and is linearly related to the area under the curve formed by a specific operating point.

This is discussed in detail in a report of mine I have uploaded, with the graphs I've discussed shown in Figure 3 for some real data relating to facial expression recognition: https://www.researchgate.net/publication/273761103_What_the_F-measure_doesn%27t_measure

Discussion of ROC Area Under the Curve is here: https://www.researchgate.net/publication/261155937_The_problem_of_Area_Under_the_Curve?ev=pub_cit

Answer 2

Precision-Recall curve and ROC curve (doesn't matter they are just the mirror images of each other) are used to give you the sense of the quality of the binary classifier for the different values for some parameter that affects the performance of your classifier. Now, F1 are particular scores which combine both precision and recall into a single one, so that way you just need to select the configuration of your classifier which has the highest F score.

In your place for each pair of precision and recall I would calculate F score and then pick the configuration which has the highest F score.

Now, the tricky part is which F score. F1 is the score which values precision and recall the same, but sometimes the recall is more important than precision (for example, you don't mind having a lot of people falsely tested for some cancer if you know that all of the ones who have that cancer are tested). In that case you could use F2 measure.

I think it doesn't make sense to sum up all F measures for all combinations of precision and recall. After all, the idea is to pick a single model out of the broader range of models, I would prefer to pick a model with the highest value of F score instead of the one with the biggest sum of all F scores.