3

The "Out-of-sample $R^2$" is defined as: $$ R^2_{OOS} = 1 - \frac{\sum_{t=\tau}^T\left(Y_t - \hat{Y}_{t\vert t-1}\right)^2}{\sum_{t=\tau}^T\left(Y_t - \hat{\mu}_{t\vert t-1}\right)^2} $$

Where $\hat{Y}_{t\vert t-1}$ is the model-ased forecast of the observation at time $t$ using data up to $t-1$, and $\hat{\mu}_{t\vert t-1}$ is the sample mean using data up tp $t-1$ (that is, it is the simple mean only forecast). Finally, $\tau$ is the starting point of the forecasting exercise. Here, $t = \tau$ is $06-1996$ and $t-1$ is $05-1996$.

I have two questions:

1). How do I calculate the $\hat{\mu}_{t\vert t-1}$ term here? I've taken an expanding average of the sample data. This is consistent with $\hat{Y}_{t\vert t-1}$, which is a rolling forecast for each year with a training set of $t-1$ observations. Is this okay?

2). My in-sample $R^2$ is $0.02$ and out of sample $R^2$ is $-0.03$. What do I do? The predictability of the 6 regressors I've been given is really low, but is this okay? Should I use squared regressors to improve the $R^2_{OOS}$ value? A negative $R^2_{OOS}$ is scaring me.

Avraham
  • 3,737
  • 25
  • 43
Alya
  • 31

1 Answers1

3
  1. That seems consistent with the approach of Campbell and Thompson, discussed here, with which I agree. Their philosophy is that, in the absence of knowing any covariate information, the best predictor of the mean is the historical mean. This could make sense as a “must beat” model.

(Campbell and Thompson use the entire history to calculate the mean. I find this to make sense.)

  1. Your value below zero indicates that the numerator is bigger than the denominator. That is, your model predictions lead to a higher value of mean squared error than you get from the “must beat” model. I view this as meaning that your model predictions are quite poor, as a simple model is doing a better job.
Dave
  • 62,186
  • The OP uses out-of-sample R-squared for time series data. Knowing how exactly the out-of-sample logic is applied/created is essential, I think. – Michael M Dec 25 '23 at 12:49
  • How does the data being a time series matter here?

    And this is part of an assignment so the regressors are given to us. I used the BIC criteria to get the model (all regressors were selected) and then found the negative OOS Rsq. I have 6 regressors, do you think taking their cross product would help?

     – Alya Dec 26 '23 at 01:39