0

What are good sign of fit from result of forecast::accuracy.

How to interpret

                  ME          RMSE      MAE       MPE     MAPE     MASE      ACF1

Training set -2.055155e-16 5.764161 4.322594 -8.302648 17.98444 6.244566 0.8651557

Test set      1.038893e+00 5.857035 4.353372 -4.400336 16.60394 6.289029        NA
Stephan Kolassa
  • 123,354
Shiv
  • 141
  • 1
    Pass on this, but a bad sign is any report implying that we can and should be thinking about 7 significant figures. – Nick Cox Oct 29 '15 at 17:06
  • I guess most statistically minded people know two or three of these immediately from the abbreviations. Perhaps only forecasting experts will know all of them at first sight. I suppose that doesn't matter because it is the latter who should be able to produce good answers. But FWIW I flag the need to explain these to any but a highly trained readership. – Nick Cox Oct 29 '15 at 17:10
  • @NickCox: re your first comment, I agree that seven sig figs are too many, but believe me - the author of the forecast package knows what he's doing. – Stephan Kolassa Oct 29 '15 at 20:09
  • MASE is one of the most extremely confusing error metric, very difficult to explain to a non technical audience, use mean absolute error if you are comparing similar measure of units. MEan absolute percentage error (MAPE) or symmetric mean absolute percentage error (sMAPE) if you have different measure of units. – forecaster Oct 30 '15 at 02:04
  • I see no justification of using MASE, unless you have zeros in your forecast which in real world forecasting is almost non existent except intermittent demand forecasting. – forecaster Oct 30 '15 at 02:12
  • I'd rather not get into a discussion with @forecaster about the merits or not of the MASE. For anyone that's interested, here is the original paper in which Hyndman & Koehler (International Journal of Forecasting, 2006) proposed it. It has gained traction in the academic forecasting community, though. And when you have zeros in your data, that's often an indication that you should not be using MASE. – Stephan Kolassa Oct 30 '15 at 11:18
  • @StephanKolassa - re: "when you have zeros in your data, that's often an indication that you should not be using MASE." What am I missing? I thought per Hyndman that MASE specifically worked well in the case of intermittent demand forecasting, as opposed to say MAPE which would blow up. So why should MASE not be used in the case of zeros in the data? – BigBen Oct 24 '23 at 14:29
  • 1
    @BigBen: that is an extremely good question. First, I'll assume that zeros are not erroneous or abnormal (in which case you should not be calculating accuracy off them in the first place). So consider intermittent demand, which is count data with "many" zeros. Here is the problem: the MASE is a scalar multiple of the absolute error, and you minimize the AE by forecasting the conditional median. If you have more than half zeros, then your MASE-optimal forecast is a flat zero. ... – Stephan Kolassa Oct 24 '23 at 15:29
  • 1
    @BigBen: ... This is presumably not what you want, which is why you probably should not be using the MASE in the first place. Turn this around: you should first figure out which functional of the unknown future distribution you want to elicit, and only then decide on your error measure. Want the conditional mean? Use the MSE (or a variation thereof). Want the conditional median? Use the MAE (or the MASE). And so forth. The argument in this thread is absolutely similar. ... – Stephan Kolassa Oct 24 '23 at 15:31
  • 1
    @BigBen: ... See also this paper and this paper for more on this argument. Feel free to contact me on LinkedIn or ResearchGate for the papers if you are interested. – Stephan Kolassa Oct 24 '23 at 15:34
  • 1
    @StephanKolassa got it! I had already read the first paper (100% agree) and thanks for the second. Thank you for your thorough work in this regard. – BigBen Oct 24 '23 at 15:48

1 Answers1

5

A MASE (Mean Absolute Scaled Error) of 6.24 in-sample is indeed a bit disconcerting. It means that your forecasting method yields in-sample absolute errors that are 6.24 times as large as those of a naive random walk model. This should not happen, unless you have a badly misspecified model.

This earlier thread on interpreting the MASE may be helpful.

In general, it is very hard to say whether a given error is "good enough" in forecasting. External benchmarks are pretty much useless, as there is just too much variation between series. I'd recommend that you simply try various approaches that model obvious structure in your data - if your series is obviously seasonal, a non-seasonal method won't be very helpful, and so on.

Community
  • 1
Stephan Kolassa
  • 123,354
  • 1
    It does seem odd to see an MASE of 6. I note in the R forecast package documentation (of accuracy) that by default MASE uses a seasonal naive forecast for seasonal series. So it could be that the model fit is handling seasonality very incorrectly. – zbicyclist Oct 29 '15 at 20:22
  • MASE is unnecessary complication in this instance, why not MAE or sMAPE? Seems to be both are very apt for this type of problem. – forecaster Oct 30 '15 at 02:16
  • @forecaster: if you have a single series, then I agree that the MASE is less informative than the MAE, since the MASE is simply the MAE scaled by a factor that does not depend on the forecast (namely, the in-sample naive forecast MAE). The MASE makes sense once you have multiple series on different levels, where you can't very well compare "raw" MAEs. – Stephan Kolassa Oct 30 '15 at 07:30