2

Let's say I have some data, and a model X that models the data. Let's say that model X has a RMSE value of 1500 and a MAE value of 200.

I build a new model Y, which is better than model X. Model Y has better error metrics: it has a RMSE of 1000, and a MAE of 150.

It is clear to me that model Y is a better fit than model X. But the question is: how much better?

I can calculate the percentage difference in RMSE like so: $ ({RMSE}_x - {RMSE}_y) /{RMSE}_y $. Can I say that model Y is 33% better than model X? Does this mean that my predictions are now 33% more accurate? If model Y is 33% better than model X, and we currently use model X to make decisions that results in \$100 profit, is it right to say that model Y will make \$133?

I suspect the answer is no to the above, given I could do the same to the MAE and get a different result. I understand that these questions are subjective, but if I have to quantify the improvement in model Y over model X, how would I do it?

Dave
  • 62,186
Tom Roth
  • 133
  • 7
  • 3
    There's no one way to do this, nor should there be. A search for a definitive meaning for the unqualified phrase "model X is p% better than model Y" is not worth your time. You simply need to state what you measure and how you measured it. – Matthew Drury Jan 22 '18 at 05:46

2 Answers2

0
  1. Model Y is 33% better than X on RMSE as a criteria is the only inference you can draw.
  2. Attributing profit values to model accuracy is little far fetched. Model will generally form a small (but important) part of your value chain. But predicting profit impact is a separate problem compared to lowering RMSE of prediction.
  3. Model comparison based on only accuracy is also not a good practice since very complex model will help you reduce inaccuracy or errors arbitrarily. The following strategies are used for more robust model comparison,
    1. Measure accuracy on a held out / test set and not on original training data.
    2. Penalize models for their complexity, preferring simpler models. This is done by using some kind of regularization while fitting the model.

In summary, design a more evolved objective criteria to compare models which will factor in complexity of the model also. Read up more about akaike information criterion, regularization etc. if you don't know about it.

hssay
  • 574
  • Assuming that both models are fit on a test set and that both models are of similar complexity, is there some way that you can link model accuracy to profit? Sometimes I'm asked questions from nontechnical people like "That's great that your new model is better, but how extra money is it worth?". I can't point them towards improvements in error metrics like MAE or RMSE because 1.the metrics aren't intuitive to them and 2. it doesn't answer their question. – Tom Roth Jan 22 '18 at 20:55
0

My typical spiel about $R^2$ seems to apply here. Below, I give a standard definition of $R^2$.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

The fraction numerator is the square loss incurred by your model. The fraction denominator is the square loss incurred by a baseline model that always predicts the overall mean $\hat y$.

However, nothing says that we have to use such a model as the baseline to which we compare our performance. Sure, that might be a reasonable “must-beat” level of performance, but if we know our competitor to be able to achieve some level of performance and need to be able to do better than that to get any business, then that sure sounds “must beat” performance.

In fact, when you run a chunk test of nested models (such as a fairly routine ANCOVA test of a categorical variable), you are comparing the performance of your full model to a model that is reduced but perhaps not reduced all the way to predicting the same value every time, so this idea even seems to exist in the canon of standard statistical analysis!

While I would have major reservations about calling such a statistic $R^2$, it seems to be quite aligned with standard statistics ideas to calculate as below:

$$ 1-\dfrac{\text{ Performance of your model }}{\text{ Benchmark performance }} $$

I would use a similar argument about reduction in error rate (reduction in MSE, reduction in MAE, etc) as I discuss here.

Dave
  • 62,186