2

MAPE (Mean Absolute Percentage Error) and SMAPE (Symmetric Mean Absolute Percentage Error) both are sensitive when the TRUE values is 0.

Thus I've come to the conclusion to add some epsilon to all data points to eliminate having zero in determinator.

But after that, I put attention that the resulting SMAPE or MAPE significantly depends on such epsilon.

For example, if I choose $\epsilon=0.1$ the MAPE is much lower, than if I choose $\epsilon=0.01$.

What should be done?

Stephan Kolassa
  • 123,354
Michael D
  • 725

1 Answers1

1

I would recommend not using the MAPE at all if your data contains zeros. The sMAPE is not much better (it is at least defined, but it contributes a fixed value of 200% each time the actual is zero, regardless of the forecast, which is rarely useful). If at all, consider using the wMAPE, which is the MAD scaled by the mean of the actuals. And even then, all these measures will almost certainly pull you towards biased forecasts. Better to use scaled versions of the RMSE. See also What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

Stephan Kolassa
  • 123,354
  • Scaled RMSE, could be for example RMSE, divided by mean of actuals? In such a case, does it preserve shift invariance? – Michael D Oct 27 '19 at 11:58
  • No, it won't, if by shift invariance you mean that adding an offset to (1) all actuals, or to (2) all actuals and all forecasts, will preserve the error measure. Measures like the MAE or MSE will be "shift invariant" in the sense of (2), but no useful error measure will be "shift invariant" in the sense of (1). – Stephan Kolassa Oct 28 '19 at 07:47
  • 1
    Also, no percentage based error measure can ever be "shift invariant" in the sense of (2), so if this is important to you, you will have to dispense with the percentages. Which means that errors (e.g., MAE or MSE) measured on different time series with different orders of magnitude will not be comparable any more. – Stephan Kolassa Oct 28 '19 at 07:48
  • MAE error by the divided by the mean of actuals could be a solution? Or a kind least worst option? – Michael D Nov 10 '19 at 10:36
  • 1
    That's known as a "weighted MAPE", see Kolassa & Schütz, 2007, Foresight. It at least avoids the problems of dividing by zero (unless all your actuals are zero), but as a scaled MAE, it is minimized by the median of the future distribution, not the mean, so you may get strongly biased forecasts if you minimize the wMAPE. For intermittent data, the wMAPE-"optimal" forecast may be a flat zero (Kolassa, 2016, IJF). – Stephan Kolassa Nov 10 '19 at 11:27
  • Just got it, that MAD, is another name for MAE. The wikipedia article was misleading: https://en.wikipedia.org/wiki/Average_absolute_deviation – Michael D Nov 13 '19 at 09:38