6

I started evaluating and comparing some methods in forecasting. I used Price of dozen eggs in US, 1900–1993, in constant dollars in the R software FMA package. I held out the last 10 years for assessment of forecast. Below are the results:

I used auto arima method in the R software. Obviously the results are way off. Am I doing something incorrect ? Below is the forecast. It does not recognize the declining trend.

auto arima

I also used an unobserved components model (UCM) and obtained a good forecast, as below.

  1. Without outliers/level shifts there are very large standard errors and therefore wide confidence bands. UCM without outliers level shifts
  2. After some iterative work, below is the output with outliers/level shifts (I know I'm overfitting here) but it did a pretty good job in forecasting; there are also narrow confidence bands. UCM with outliers level shifts

In looking at just this example the UCM seems to predict the hold-out sample more accurately than auto.arima.

Why is auto.arima not providing a reasonable forecast?

Are state space models/UCMs better for forecasting long range?

Are there any benefits of using one method over other?

Richard Hardy
  • 67,272
forecaster
  • 8,145

3 Answers3

9

A recent change to the way regression coefficients are initialized in the estimation of ARIMA models means that a different model is now selected by auto.arima (using R3.0.2):

> auto.arima(window(eggs,end=1983))

ARIMA(0,1,0) with drift         

Coefficients:
        drift
      -2.2665
s.e.   3.1133

sigma^2 estimated as 804.5:  log likelihood=-390.15
AIC=784.31   AICc=784.46   BIC=789.14

See https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=15396 for discussion of the change in the initialization. This is relevant here as the drift is estimated as a linear regression. The change is in stats::arima, not in auto.arima.

Note that the drift is not significant in any case. So you can hardly claim that the non-trended model is unreasonable.

Of course, if you know something about the data, then you should use the knowledge you have. But if you want something completely automatic, you can't really complain if it gives you something that doesn't fit your preconceived ideas of what is reasonable.

A state space unobserved component model will be very similar to an ARIMA model for these data. In fact, there will be an equivalent ARIMA model for any sensible UCM fitted to the data. So it makes no sense to say one forecasts better than the other. One might be easier to use than the other, or more interpretable than the other.

Yes, there are benefits. ARIMA is much better at easily handling complicated short-term dynamics. UCM is much better at decomposing the series into interpretable components.

Rob Hyndman
  • 56,782
  • Dr. Hyndman, I am not sure if there was a change in the auto.arima function or not, but you can see in my answer that indeed the function picks up the downward trend. Did you mean to place d=0 in your last code snippet? That seems to actually produce an UPWARD trend. – B_Miner Jan 25 '14 at 23:36
  • I don't think anything has changed in auto.arima() itself to cause this change in behaviour. What did change a few months ago was the initialization used in the optimization algorithm in stats::arima which is called by auto.arima(). See https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=15396 for details. – Rob Hyndman Jan 26 '14 at 22:57
  • My answer now updated. – Rob Hyndman Jan 26 '14 at 23:09
3

I employed AUTOBOX ( a piece of software that I have helped develop ) . The automatic model identification scheme detect a first difference model with an ar(3) component.enter image description here . The test for constancy of parameters revealed a possible breakpoint at or around period 53 (year=1953 note that the UCM model declared a new trend at 1954) which suggests a possible regime change between 1-52 AND 53-83 . This is easily confirmed visually by examining a plot of the data. It is an exponential smoothing model ( a very particuLlar case of an ARIMA MODEL ) with a constant and an adjustment for the 75th data point. The residual plot is suggestive of sufficiency (at least with 31 values) enter image description here with an ACF of enter image description here . The next plot is the actuals/fit and forecast . enter image description here

The ARIMA model identified has a significant negative constant and thus provides downward guidance. The difference between AUTOBOX other approaches is that AUTOBOX tested for transient parameters and concluded that the older data (obs 1-52) was inconsistent with observations 53-83. This is the "elephant in the room" that nobody dares to mention. enter image description here The assumption that all of the data comes from the same model with constant parameters needs to be verified NOT ignored. Just because we know that 83 values exist DOESN'T mean that we should use all of the data. Modelling the entire series does not necessarily model individual subsets.

I must comment that evaluating a forecast from 1 origin is insufficient research . One must look at many origins and for possibly different lead times.

IrishStat
  • 29,661
  • Thank you, this is very helpful. Your approach captured the downward trend part. I agree with your comment on looking at multiple origins. I was just looking at this example where auto.arima did not work in my opinion. I'm intrested to know why auto.arima did not capture the downward trend while UCM and autobox where able to capture it ? – forecaster Aug 27 '13 at 22:53
  • 2
    @forecasterThe model identifcation strategy revolving around using the AIC has in my opinion a number of failures. The first failure is that if the series contains Pulses/Level Shifts/Seasonal PUlses or Local Time Trends these will obfuscate correct ARIMA identification via AIC. The second flaw in my opinion is the presumption that ALL of the data should be used indiscriminately to identify and estimate the model.This data set is a "poster boy" illustrating the second flaw.Additionally if the variance of the errors is not-constant then this needs rectification. This can also cause AIC failure! – IrishStat Aug 28 '13 at 01:04
  • I can not imagine why someone would downvote this response and not leave a comment as to why. If something was incorrectly stated I would like to know what that was so I can make appropriate corrections. I guess even the collection of statisticans have their anomalies ! – IrishStat Aug 28 '13 at 13:23
  • I'm not familiar with mechanics of autobox, can you please let us know if you chose 53 - 83 manually to do the forecast or did the software automatically picked observation starting 53 and foretasted it ? – forecaster Aug 30 '13 at 02:20
  • @forecaster It did it automatically by searching for alternativre break points and then tested each of the possible breakpoints via the CHOW test – IrishStat Sep 03 '13 at 12:46
1

I don't get the same result as the OP (version 5.0 of the forecast package). If you run the following, the result is indeed a linear downward trend.

install.packages("fma")
library(fma)
install.packages("forecast")
library(forecast)

#window as per the OP
eggs2<-window(eggs, start=c(1900), end=c(1983))   
plot(eggs2)

#does produce a linear trend downward!
model2<-auto.arima(eggs2)
model2
plot(forecast(model2,10))

Maybe there was a change where allowdrift was not TRUE by default in the package used?

enter image description here

Further, if you run auto.arima on the full data set (without withholding the last 10 years) which is I believe what Dr. Hyndman was doing, the down ward trend is picked up.

model<-auto.arima(eggs)
model

Series: eggs 
ARIMA(0,1,1) with drift         

Coefficients:
          ma1    drift
      -0.1630  -2.3774
s.e.   0.1145   2.3229

sigma^2 estimated as 713:  log likelihood=-432.26
AIC=870.51   AICc=870.78   BIC=878.11
B_Miner
  • 8,630