0

I have two sets of forecasting errors, and want to perform a DM test.

Both forecasts are a fixed size moving window, and are 1 day ahead forecasts.

The first step of performing the DM test is to calculate the difference in loss functions, which I have done. As I understand it, this could be autocorrelated, which affects the standard errors.

Whilst I can calculate standard errors like this: s.e. = sqrt(var(dt)/length(dt)), I believe that, if there is autocorrelation, I need to calculate the standard errors like this: s.e. = sqrt((var(dt)+2*sum(cov))/length(dt)), where cov is a vector of k autocovariances. How do I choose k, which are the number of autocovariance lags? Is there a test to conduct? I have heard about using BIC to calculate this.. How do I do that? I have plotted the autocorrelations and autocovariances with the acf function, but still don't know what to do from here. I have over 1100 forecasted observations if that helps.

if the dm statistic is statistic = (mean(dt) / s.e.), I essentially want to know how the s.e. is calculated.

Guidance is much appreciated!

Richard Hardy
  • 67,272
TheManR
  • 93
  • The primary reason for autocorrelation in forecast errors is forecast horizon being more than 1. In your case it is equal to 1, so it should not be a problem. See how this is done in the dm.test function in the forecast package in R. It follows the paper by Harvey et al. (1997) referenced to in the function's help file. – Richard Hardy Nov 01 '18 at 10:23
  • Is this true for loss functions other than MSE though? For example, the quasi likelihood loss function (QLIKE)? I have already checked this function in the forecast package, however, I do not like the way it functions, in particular I have no idea why it references variance from the ACF (seeing as I don't know what the acf is at lag = 0). If you have any idea what acf when lag.max = 0 produces, please educate me! It is clearly not the variance of what is being tested, but seems close. Additionally, I do not like the function using the small sample adjustment by default – TheManR Nov 01 '18 at 11:53
  • ACF at lag 0 is precisely the conventional estimate of variance. It the the covariance of the series with itself, hence, variance. However, in autocorrelated time series, the conventional estimate is biased, so information from the ACF for other lags is used to adjust it upwards to make it unbiased. Regarding the small sample adjustment, what are youe referring to? Normally, these adjustments tend to be justified. In large samples, first-order approximation is good enough, while in small sample, an adjustment is made to use finer approximation. This is perfectly valid for large samples, too. – Richard Hardy Nov 01 '18 at 12:20
  • is there a reason it provides a different variance than the var() function? Much appreciated! – TheManR Nov 01 '18 at 12:22
  • I suggest you look up some threads discussing variance estimation for time series (as opposed to i.i.d. or cross-sectional) data. Long-run variance or unconditional variance is what you should be looking for. – Richard Hardy Nov 01 '18 at 12:24
  • This is one such thread on long-run variance. – Richard Hardy Nov 02 '18 at 07:57
  • I forgot to mention, I am forecasting volatility, and thus there would be some autocorrelation due to the nature of volatility – TheManR Nov 14 '18 at 08:29

0 Answers0