0

I trained an ARMA model using Python's from statsmodels.tsa.arima.model import ARIMA .

Separated the training and testing data, and fitted the model with parameters p, d, q = 1, 0, 1.

Below is the result of plotting my rolling prediction results (red) against the test (orange). At first, I thought this was not making good predictions.

enter image description here

However, when I simply plotted the prediction results alone, I got something pretty good looking (below). enter image description here

This is not the best but I think it's at least catching some trends. So I tried simply scaling it up by a constant multiplier and a bias by eyeballing. Then this happened (last picture).

enter image description here

What is going on? I expected something like the last picture for the forecasted predictions, but somehow all the predicted values are significantly attenuated.

Why would this happen? Why would I get a seemingly good prediction when I randomly scale up and add a bias? Am I getting lucky? How can I do this more systematically?

Richard Hardy
  • 67,272
js9
  • 125
  • Any particular reason why you select your own ARIMA(1,0,1) order, rather than having well established tools like pmdarima do so for you? – Stephan Kolassa Nov 01 '23 at 06:23
  • I checked autocorrelation graphs and observed the peak – js9 Nov 01 '23 at 21:19
  • 1
    Ah. I would very much encourage you to read this thread. You actually can't identify an ARMA(p,q) model based on the ACF and PACF plots if both p and q are nonzero. It is much better to rely on proven tools, which typically perform a grid search using information criteria. – Stephan Kolassa Nov 01 '23 at 21:27

1 Answers1

2

The proposed duplicate is not exact, but close enough. ARIMA, like any other forecasting method, tries to separate signal (forecastable) from noise (not forecastable). Usually, most of the variation comes from noise, therefore the forecast will vary much less than the historical data. This is exactly as it should be. If your data generating process follows an ARIMA(p,d,q) process and you have "enough" data, then an ARIMA model of the same order will give optimal (i.e., MSE-minimal) forecasts, even though the forecasts vary less than the history.

Your proposed solution of scaling forecasts does not work, because it relies on getting a scaling factor right. In your example, you do that by looking at the holdout actuals, but now they are not holdout any more. You can't do that for "real" forecasts without waiting for the actuals to come in - by which time you don't need a forecast any more. And no, unless you know a lot of highly specific information about your time series, you won't be able to integrate such a scaling into your forecasting workflow. (And if you can do so, more power to you, but it's not an ARIMA forecast any more.)

You may be interested in our resources on forecasting here: Resources/books for project on forecasting models

Stephan Kolassa
  • 123,354
  • 1
    One more reason why scaling will not work well is that the scaling you get by default is close to optimal. Using a different scaling will usually be further away from optimal and will thus increase the forecast errors. Perhaps worth mentioning. – Richard Hardy Nov 01 '23 at 07:53
  • Does it mean that when ARIMA works well (generating very close shapes and trends) the variation is not driven much by the noise? – js9 Nov 01 '23 at 21:18
  • Do you mean in-sample or out of sample? ARIMA typically matches the patterns very well in-sample (shifted by a period or so), but looks dramatically less noisy out of sample. In-sample accuracy is very misleading as an indicator of out of sample forecast accuracy. But if your ARIMA gives you a good match out of sample, I would say that indicates low residual noise. – Stephan Kolassa Nov 01 '23 at 21:25
  • I meant out of sample like the picture I attached. I tried to follow some tutorials online (https://www.kdnuggets.com/2023/08/times-series-analysis-arima-models-python.html) and was expecting the result to look like what's shown at the end. But I guess it does not work for every case... – js9 Nov 01 '23 at 22:28
  • 1
    Hm. In that thread, the plot looks like this because it contains rolling 1-step-ahead forecasts: after each time step, they refit the model, forecast out one step, collect that single forecast, then advance time by one step. This forecast is very "adaptive". I do not think you are doing that here, are you? If you did, your forecast plot should look different. – Stephan Kolassa Nov 02 '23 at 05:46
  • I followed the post and also tried to do the rolling 1-step-ahead forecasts (refit after each time step). Yet I get such a suppressed prediction, which I found very strange. – js9 Nov 06 '23 at 22:47