0

Which attribute should consider as best fitted model AICc or RMSE in auto.arima? I am having a case of

best$aicc <- Inf 
for(i in 1:25){

r = fourier(timeSeries,K=i)
ar <- auto.arima(anotherTimeseries, xreg= r, seasonal = false)
 if(ar$aicc < best$aicc){
     best = ar /* this part is reached only once for my data, the only first value is set as best and the aicc value keep on increasing*/
 }else{
    acc <- accuracy(ar)
    acc1 <- accuracy(best)
    /* check the RMSE accuracy rate and the set lowest RMSE value to best*/
 }
  forecast(best, xreg = (best i value from pervious value), h= 104)
}

Now, The doubt is whether I need to choose a best fitting model based on aicc value or RMSE value check (in the else part). Which approach will be proper? 

AICC                RMSE
1642.857        acc- 233.6344
                acc1 - 234.3495

1651.623        acc- 233.3246
                acc1 - 234.3495

                acc- 232.7801
1656.273        acc1- 234.3495

RMSE value decreases in every step but AICC value increases. Which one would be better arima model ? Thanks in advance for the suggestions

Rajan
  • 105
  • 4
  • The Reason for choosing lowest AIC is clear. As well as am gonna using the model to forecast. Thanks, Richard. – Rajan Apr 25 '17 at 23:24

1 Answers1

1

forecast::accuracy will give you in-sample accuracy measures which are useless for model selection. By construction, a model with some additional Fourier terms will beat a model without them in sample. But we are normally interested in generalization performance, i.e. out of sample.

On the other hand, AICc estimates $-2n\ \times$ expected log-likelihood out of sample (as explained in this answer and can be found in Hastie et al. "The Elements of Statistical Learning") and is a sound criterion for model choice, especially if the goal is forecasting. You should pick the model with the lowest AICc.

Richard Hardy
  • 67,272