1

I am trying to fit the weekly forecasting model, however, I only have 60 data points.. trying tslm i.e. the linear model with time series components, and comparing it to harmonic regression, the linear model had lower AIC but higher AICc. As a further test, I then split my data into 54 training and left 6 as test data and refitted, and compared their performance using MAPE, TSLM gave a better performance using MAPE as a metric.

TLDR Now I'm confused about which method I should proceed with since tslm is better in aic and mape whereas harmonic regression is better in aicc. Any help is appreciated.

Basil Abu Mallooh
  • 11
  • 2

1 Answers1

0

60 data points is very little for weekly granularity - you have not even seen two years of history yet. Any seasonal pattern will be very hard to disentangle from noise.

Looking at a holdout sample is very good practice. However, again, 6 data points is very little to base any conclusions on.

Note that the MAPE elicits a different point forecast than the expectation: What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? Strictly speaking, the AIC (and its small sample version, the AICc) does not evaluate a point fit, but an entire distributional fit, since it contains the log-likelihood. Almost certainly, there is an assumption of normality in your pipeline.

So if you really want a low MAPE, I would first decide whether to trust the AIC or the AICc more. To be honest, given the small dataset you have and the fact that your model is almost certainly misspecified, I would not attach great significance to this decision. Plot your series and your predictions and eyeball them. (See below.)

Once you have decided on a model, forecast it out. For each future datapoint, simulate from the predictive distribution. Numerically optimize the point prediction that minimizes the expected APE. Per the link above, this will be lower than the expectation point forecast. (Of course, if you actually want an unbiased expectation point forecast, you can use the point forecasts your models give you, no need to simulate and optimize - but then, evaluating this using the MAPE does not make sense.)

Then again, you could also use an ensemble forecast. Your two models seem to be both reasonable, otherwise AIC and AICc would not give conflicting results. So you can proceed as above, but simulate from an equal mixture from the two predictive distributions. And again, if you really want an expectation forecast, simply take the average of the two models' expectation forecasts. (Trying to optimize weights is usually not very helpful, Claeskens et al., 2016.)

Stephan Kolassa
  • 123,354