Out-of-sample MSE and MAE for volatility forecasting

Question

I have been searching through the whole CrossValidated but couldn't find the answer.

I want to test out-of-sample the volatility forecasts (if it means something ARCH-like ones, MSGARCH, Multifractal models). I want to use MSE and MAE metric but am not sure what to take for actual volatility.

I know that probably the best I can do is to take realized volatility of that day based on the sum of the intraday returns but I don't have access to that data.

So questions are the following:

Can I compare it to squared returns as written here On forecasting, the mean squared error and realized volatility In this case my problem is that I feel like couple of outliers will influence score much more than most of the data set.
Can I use absolute returns similarly as to in the point one?
Is there any better reference than this?
Is there any access to intraday data of S&P500 for example and if not is there maybe estimated volatility from such a data available?

Reference to any such answer is very welcome

Some references: Patton & Sheppard "Evaluating Volatility and Correlation Forecasts" (2009) and Patton "Volatility forecast comparison using imperfect volatility proxies" (2011). — Richard Hardy, May 16 '21 at 16:59
Maybe "On forecasting, the mean squared error and realized volatility" can help, too. — Richard Hardy, May 16 '21 at 17:10
As far as I remember from Patton's papers, MAE does not work well, while MSE and Qlike are OK for evaluating volatility forecasts. — Richard Hardy, May 16 '21 at 17:41
Thank you, this helps a lot. According to Patton, your last comment is correct. Sorry if the following question doesn't make sense, I am fairly new to this but what I don't understand is the following: I have let's say last 365 returns of S&P500. I train the model on them and predict one step ahead volatility. During the Corona time S&P500 dropped by 12%, this means because I train on 100returns, my biggest returns^2 is around 144. My biggest predicted volatility is around 6-7. Should I have trained on returns instead of returns100 or did MSE on variance instead of sqrt of it? — Selena Pepic, May 16 '21 at 19:19
It is normal that the underlying variance varies less than realizations of a time series. You may simulate a true GARCH process with known parameter values and see for yourself. — Richard Hardy, May 16 '21 at 20:12
Sorry, one last thing to confirm. You say that I should use https://drive.google.com/file/d/1opaZfiwVbdMzBQ0QnADpT3ZZA205JGgE/view this as representative and not https://drive.google.com/file/d/1kSU5mQFJ6olzC4s2IzyTp6F6YsKVlCcX/view even though squared returns are in a way proportional to squared volatility? — Selena Pepic, May 16 '21 at 21:47
I might have misunderstood you. It makes sense to compare $r_t^2$ with $\hat\sigma_t^2$, not with $\hat\sigma_t$. — Richard Hardy, May 17 '21 at 05:20

Out-of-sample MSE and MAE for volatility forecasting

0 Answers0