Accuracy metric for comparing Time Series models?

Question

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below).

All 3 have different numbers of "input parameters": SLR just uses time, ARIMA and SARIMAX both use time and 12 lagged $y$ values. *I say "input parameters" b/c I'm not sure how to consider $y$.

I'm currently using RMSE to compare them. Is this an acceptable practice, or is there another measure I should use that takes model complexity into account (e.g. something akin to adjusted R^2)?

I know that MAPE is a commonly used forecasting metric. But like RMSE, I'm not sure it's appropriate for comparing models with different numbers of input parameters. Just wondering if there's anything better out there.

Out of sample. I'm interested in the RMSE of the forecast, plus the RMSE of the in-sample ARIMA and SARIMAX models are nearly identical (the differences b/t them only really show out-of-sample). — infinitely_improbable, Sep 04 '18 at 04:27
Rather than choosing a single measure, it is more common to include several relevant measures of accuracy for prediction. — Frans Rodenburg, Sep 04 '18 at 08:14
These kinds of adjustments for model complexity are typical of in-sample measures, but not of out-of-sample measures, which tackle overfitting in an entirely different way. You can compare RMSE/MAPE/MASE/etc out-of-sample between these models, just make sure that your exogenous regressors aren't assumed known in the future if they aren't really. — Chris Haug, Sep 04 '18 at 12:18
@ChrisHaug: do you want to post your comment as an answer? In addition, the forecasting textbooks listed at this thread all give error metrics commonly used in forecast evaluation. OP may be interested in the shortcomings of the MAPE. And $R^2$ is not commonly used in forecast evaluation. — Stephan Kolassa, Dec 08 '23 at 23:23

Accuracy metric for comparing Time Series models?

0 Answers0