6

In Patton (2011) the author finds that both the MSE and the QLIKE loss function are robust when used to compare rivalling volatility forecasting models, which means that using a proxy for volatility gives the same ranking as using the true (unobservable) volatility of an asset.

In my current project I am comparing a family of GARCH/AGARCH models and while the MSE suggests that nothing outperforms a GARCH(1,1), the QLIKE statistic suggests that an APARCH(1,1) model performs significantly better.

Is this caused by the two loss functions penalising deviations differently? Specifically, what do the two loss functions place the highest penalty on, i.e. how do I interpet this?

I am hoping this is not down to some trivial coding error.

#MSE 
MSE<-function(sigmafc,RV){
  MSE=1/length(sigmafc)*sum((sigmafc^2-RV)^2)
  return(MSE)
}

#QLIKE
QLIKE<-function(sigmafc,RV){
  varfc=sigmafc^2
  QLIKE=sum(
    (RV/varfc-log(RV/varfc)-1)
    )
  return(QLIKE)
}

I gather that the MSE depends on forecast errors, while QLIKE depends on standardised errors, but how would I interpret this?

Pedestrian
  • 111
  • Could you give a full reference for Patton (2011)? Also, where exactly in the paper is QLIKE defined? I would like to check whether your code matches that definition? (I have found some paper by Patton from 2011 and I don't think you got QLIKE right, but that is based on just a quick look at the paper.) – Richard Hardy Jul 27 '16 at 11:52
  • Patton (2011) does indeed have a different definition of the QLIKE loss function. However, in later papers (and in my lectures) the function was given as

    $QLIKE=\frac{\hat{\sigma}^2}{h}-\log{\frac{\hat{\sigma}^2}{h}}-1$ with $\hat{sigma}^2$ being the term for the volatility proxy and $h$ the forecast for a given period.

    The reasoning, if I recall correctly, is that this gives a QLIKE value of 0, if the forecast and the volatility proxy are identical.

     – Pedestrian Jul 27 '16 at 13:58
  • The paper I reference is listed below http://econpapers.repec.org/paper/utsrpaper/175.htm – Pedestrian Jul 27 '16 at 14:05
  • After a bit more searching, the reasoning is presumably that the MSE is more sensitive to outliers/extreme values – Pedestrian Jul 27 '16 at 15:14

1 Answers1

2

Just noticed this question is over two years old, but this answer might prove useful for future readers.

Check Figure 3 of the paper you referenced in the comments, or Figure 1 of the version of the paper that was published in the JoE. This figure demonstrates the shape of several different loss functions, including both MSE and QLIKE.

Importantly, you'll note that MSE and QLIKE have a very different shape, so you can expect the results from using the two loss functions to be quite different. Your assertion in the comments that MSE is more sensitive to outliers is mostly correct, except for the far left tail of the loss function, where QLIKE is actually more sensitive to outliers.

More generally, what you're running up against is one of the core problems of this area of the literature. Volatility processes in asset returns tends to be dominated by a small number of very large observations. This means that loss functions such as the MSE are not good at rejecting null hypotheses, since the analysis is typically dominated by a small number of large observations that get emphasized by the non-robust nature of the MSE loss function.

QLIKE partially solves this problem by being robust to extreme observations in the right tail, but unfortunately it is not particularly robust in the left tail. Further, since QLIKE is not a symmetric loss function (in fact MSE is the only symmetric loss function in the class discussed by Patton), so it penalizes positive and negative loss differently. This means that if you are comparing two forecast procedures, one of which on average produces positively biased forecasts, and the other on average produces negative biased forecasts where the bias is of the same magnitude, then using QLIKE will massively favour the forecast with positive bias. So unless you have some reason to particularly prefer bias of one kind over another, QLIKE must be regarded as an imperfect solution to the problem.

So what is the solution? There may not be one, unless you're willing to make some further structural assumptions about the true data-generating process behind your model.

Colin T Bowers
  • 807