2

I would like to check in percentage the accuracy of my regression model. I know that normally accuracy is used as a metric for classification. I have evaluated my model based on r-squared and also plotted the y_pred versus y_test.

However, I find it easier to understand the prediction performance in terms of accuracy. Therefore, I would like to do something like this (based on the sklearn, metrics.mean_absolute_percentage_error) :

accuracy = 100 - np.mean(mean_absolute_percentage_error(y_test,y_pred))
print('Accuracy:', round(accuracy, 2), '%.')

Does it make sense, would the result reflect the performance of the regression model based on a percentage of accuracy?

Monica Pena
  • 23
  • Are you sure of how to interpret such a score? For instance, do you know that a percentage of $50%$ is like an $\text{F}$-grade in school that indicates a poor model? Do you know if $90%$ is like an $\text{A}$-grade in school that indicates an excellent model? // I disagree with this being a duplicate of the MAPE question, though that information is pertinent. – Dave Nov 03 '21 at 11:01
  • @Dave, I understand the analogy, however, I find it easier to understand in percentage terms rather than an r-squared of 0.4 (for instance).

    It does not mean that the model is performing better or worse, it could be still pretty bad, just a different way to communicate the same result.

     – Monica Pena Nov 03 '21 at 11:08
  • Then why not take $R^2=0.4$ as $40%$ performance? – Dave Nov 03 '21 at 11:17

1 Answers1

2

$R^2$ and MAPE do not communicate the same result. $R^2$ elicits the conditional expectation, and MAPE elicits the (-1)-median. Those are two different functionals of the underlying distribution.

In my opinion (Kolassa, 2020), you should first think about which "one number summary" of your predictive distribution you would consider "good", and then use an error measure that is optimized in expectation by that functional. If you want to predict the conditional expectation, use the MSE. If you want the conditional median, use the MAE. If you want the conditional (-1)-median, use the MAPE.

As a corollary, your model very likely does not optimize on MAPE. I argue (again Kolassa, 2020) that it makes no sense to run, say, an OLS and calculate point predictions (which are expectation predictions, which minimize the MSE in expectation), and then evaluate these predictions using a different evaluation measure. Instead, if you want MAPE-optimal predictions, use the MAPE as an objective function in your fitting step, as well - similar to how quantile regression uses a quantile loss to predict conditional quantiles.

Stephan Kolassa
  • 123,354