Text classification - High Accuracy, low recall and low precision

Question

I am using fastai to create a text classifier that labels texts as either 0 or 1.

My data (number of 1's and 0's) for training is balanced, and I got an accuracy of 85%.

To test, I used a new unseen corpus of data - to try an mimic a real world scenario, that is not balanced, and produced a confusion matrix.

According to my confusion matrix, my precision and recall are really low at around 20 and 14 percent respectivly.

What are possible reasons for that ? What Can I do to improve these metrics ?

For that to happen, low precision and recall, you model would have to consistently classify in the opposite class. Are you sure the labels are coded right? — user2974951, Sep 05 '19 at 08:44
Related: Why is accuracy not the best measure for assessing classification models? Everything discussed there also applies to precision and recall. — Stephan Kolassa, Sep 05 '19 at 08:45
@user2974951, I labelled the data myself. I have 1000 1's and 1000 0's. The only way the data differs is in the length of each text. Could that be an attribute ? Is that what you mean ? — noor h, Sep 05 '19 at 09:16
@StephanKolassa Thank you. If I understand correctly, using sensitivity and specifity is a better metric? — noor h, Sep 05 '19 at 09:43
I would argue that you should build a probabilistic classifier and evaluate that using proper scoring rules. See also the links in my earlier answer. Sensitivity and specificity suffer from the very same problems as accuracy, precision and recall. — Stephan Kolassa, Sep 05 '19 at 09:47
@StephanKolassa, I understand! Thank you very much for your input — noor h, Sep 06 '19 at 04:15

0 Answers0