1

The problem I'm working on is a multiclass-classification. Have been reading through lot of articles and documentation, but not able to figure out which of Accuracy_Score or Cross_Val_Score should be used to find the prediction accuracy of a model.

I tried using both but the scores are different. Cross_Val_Score() gave me 71% right prediction, but 69.93% using Accuracy_Score().

enter image description here

enter image description here

What could be the possible reason for mismatch?

Edit: Added Confusion Matrix and Classification report: enter image description here

ranit.b
  • 403

1 Answers1

1

They're going to be different because in cross_val_score, you obtain an accuracy for each of your folds and average them. For each CV fold, your training and tests set are different; so, you obtain different accuracy values for each of them, and it enables you to calculate standard deviation of your accuracies, which is enclosed in parentheses in your image. accuracy_score of sklearn.metrics library calculates the accuracy based on the inputs y_pred and y_true. For example, if you input your entire training set, you'll get accuracy of your entire training set, which is of course slightly different than your CV score.

gunes
  • 57,205
  • Gunes, Thanks for the clarification. I see your point now. So, can we say accuracy_score is the correct way to measure overall prediction accuracy? – ranit.b Feb 23 '19 at 14:23
  • It actually depends on what you're trying to measure, all metrics have some meaning, but for final evaluation of how good your model is, use accuracy_score with your test set. (Assuming you're not into precision/recall, or confusion matrix) – gunes Feb 23 '19 at 14:26
  • Why do you say "not into precision/recall, or confusion matrix"? Yes, check them as well. Note, i've just added the confusion matrix and classification report. – ranit.b Feb 23 '19 at 14:29
  • Because my comment could be misinterpreted for evaluating the model performance. It's highly valuable to check other metrics, especially in imbalanced datasets. – gunes Feb 23 '19 at 14:32
  • Isn't model accuracy same as model performance? – ranit.b Feb 23 '19 at 18:35
  • Model performance is more general, and depends on your business case. Accuracy, F1 score, AUC in binary cases, e.g. are all metrics you can use to evaluate the model performance. In a binary classification problem with 99-1 class (class 0-1 respectively) split, one can easily achieve 99 % accuracy by classifying all the samples as class 0. But, is your model performance really that good? – gunes Feb 24 '19 at 06:47
  • In your example, if model accuracy is 99%, then we can say model performance is good. But, in my case, since accuracy is 69.9%, the performance is not good. Is this understanding correct? – ranit.b Feb 24 '19 at 09:54
  • 1
    No, in my example model performance is not good. Because, typically, in datasets with imbalance, people don't use accuracy for measuring performances. For your case, I wouldn't look at the accuracy directly, since your precision for class 3 is 0. There are more serious problems. And, I advise you to read through the answer here for a discussion on accuracy: https://stats.stackexchange.com/questions/312780/why-is-accuracy-not-the-best-measure-for-assessing-classification-models, because this seems to be going towards another direction other than the main question posed here. – gunes Feb 24 '19 at 10:03
  • True Gunes. The link you posted seems to be a good pointer. Thanks! – ranit.b Feb 24 '19 at 10:04