on the use of R squared as a measure of predictive accuracy for non-parametric models such as random forest

Question

I am confused. I know there are a couple of similar questions about $R^2$ but I hope I get some opinions on this particular matter.

I have trained a random forest and other nonparametric regression models and I want to test their performance on unseen data. I want to measure their predictive accuracy.

I am an engineering student which is not particularly good at statistics. I know we must differentiate between measuring goodness of fit (GoF) and predictive accuracy. The difference being the former is measured on the training data and the latter on test data. But it does not mean that we must have different metrics for each. Correct me if I'm wrong, please.

I have read some references on the fact that $R^2$ should not be used for measuring GoF if our model is not linear or cant be transformed somehow to a linear model (Kvålseth, 1985) (Spiess and Neumeyer, 2010).

Now you may ask, which definition of $R^2$. Thats part of the confusion too. Lets take the most common ones:

$$ R_1^2 = 1 - \dfrac{\Sigma (y_{true} - y_{pred})^2}{\Sigma (y_{true} - \bar y_{true})^2}$$

The above version is the one that is used in the popular scikit-learn package in Python.

And $R_2^2$ being the squared correlation coefficient (Pearson's $R$). This one is used in the caret package in R.

The interpretation for both of them: The proportion of total variance of $y_{true}$ as explained by the fitted model.

Two things I gather from this:

1. It is apprently only a measure for GoF
2. Since it is a proportion, it is meaningless to be negative and MUST be between zero and one.

I want your opinion on this: in my field (hydrology) researchers use Nash–Sutcliffe efficiency (NSE) score, which is exactly calculated as $R_1^2$, as a way to measure predictive accuracy or power of a hydrological models which clearly are not linear. Their rational is that the model should do better than the benchmark, their benchmark being the $\bar y_{true}$. Therefore negative values of NSE means that our model is doing worse than the mean target.I have a feeling that this is fundamentally wrong. This benchmark estimator is vague and how we can have it on unseen data to being with? and also since NSE is basically $R_1^2$ we can not use it as a measure of predictive accuracy.

Now my questions:

1. should/can I use $R_1^2$ to measure accuracy of my predictions and random forest?
2. Can I use $R_2^2$ for the above-mentioned purpose?
3. Besides metrics like MAE and RMSE, what are other options to qualify the performance of non parametric models on test data? in terms of accuracy or association,

Here is a subset of my test data prediction and observations:

\begin{array}{|c|c|} \hline {} & y\_true & y\_preds \\ \hline 0 & 3.745821 & 4.894624 \\ \hline 1 & 3.940449 & 5.743571 \\ \hline 2 & 2.849447 & 4.726890 \\ \hline 3 & 1.653091 & 2.659571 \\ \hline 4 & 2.934447 & 4.244686 \\ \hline 5 & 3.346146 & 5.269689 \\ \hline 6 & 2.450010 & 4.651610 \\ \hline 7 & 3.393356 & 5.122578 \\ \hline 8 & 0.791639 & 1.656736 \\ \hline 9 & 0.893791 & 1.935156 \\ \hline 10 & 0.129959 & 3.976739 \\ \hline 11 & 2.043000 & 4.072408 \\ \hline 12 & 4.298383 & 4.357470 \\ \hline 13 & 3.115428 & 4.432231 \\ \hline 14 & 4.325494 & 4.599493 \\ \hline \end{array}

(The values are volume of daily evapotranspiration in mm)

for this subset and my random forest:

$R_1^2 = -0.87$ and $R_2^2 = 0.55$.

Answer 1

As you write, the only difference between Goodness of Fit and predictive accuracy is that one is measured in-sample and the other out-pf-sample. The terms themselves say nothing about the KPI used. So you can use $R_1^2$ or $R_2^2$ in either case, or any other accuracy measure. (It would make little sense to use a pinball loss that is optimized by a quantile for the GoF, though.)

Whether the models are linear or nonlinear is immaterial for the accuracy KPI, which after all only assesses how accurate predictions are. We can evaluate predictions from judgmental forecasts with the exact same KPIs, after all.

To your questions:

1. As Dave writes, $R_1^2$ is equivalent to MSE: it calculates the MSE, divides it by a scaling factor which is just the sum of squared differences between the actuals and their mean, and then subtracts this from 1. In-sample, this is usually a positive quantity, because your model should fit at least as well as the overall mean. (If it does worse in-sample, you have probably made an error.) Out-of-sample, it is quite possible that the mean of the holdout actuals would have been a better prediction than the prediction you calculated, see here. In such a case, your "out-of-sample $R_1^2$" would be negative.

   So yes, you can use $R_1^2$ to evaluate predictions.

2. I personally would not use the squared correlation $R_2^2$, because it does not account for bias. If $\hat{y}_i=y_i-3$ for all $i$, then your predictions are biased, but they correlate perfectly with the actuals, and $R_2^2=1$. So $R_2^2$ will not detect that this prediction is likely not one you would consider "good".

3. There are many, many accuracy measures for predictions. Besides the MSE and $R^2$, there is also the MAE, the Mean Absolute Percentage Error, various pinball losses for quantile predictions and so forth. The key thing is that you need to think about which functional of the (unknown) future distribution you want to elicit, and then use an appropriate error measure. Do you want unbiased predictions? Use the MSE. Do you want the median? Use the MAE. Do you want a quantile prediction? Use a pinball loss. You may be interested in Kolassa (2020, "Why the "best" point forecast depends on the error or accuracy measure", International Journal of Forecasting), where I explain this in a little more detail.