Selecting the best model using cross-validation on coefficient of determination and/or mean squared error

Question

Let's say I'm doing regression and I want to know whether I should use Linear Regression or Random Forests. I do 10-fold cross validation on each model to obtain scores of $R^2$ and Mean Squared Error (MSE).

How do I then compare these lists of numbers to see which is better? Do I choose the one that has the highest mean $R^2$? Or what about the lowest mean MSE? Or maybe the highest minimum $R^2$, or the lowest maximum MSE, or some weighted average of these quantities? What if the $R^2$ is better on one but the MSE is better on another? Is it arbitrary? Does it depend entirely on the particular problem I'm trying to solve?

Is there a generally accepted silver bullet?

Answer 1

I would say that the answer should not come from the data, but rather from the science producing the data, the question to be answered, and the audience that you are trying to answer the question for.

Think about the science that underlies the data/question. Do the discontinuities in the Random forest model make sense? are you allowing enough interaction and non-linearity in the regression model?

What will your audience best understand? (random forests can sometimes be converted into a simple lookup table without needing to do any math, regression models can be more straight forward for future computation)

These and other questions outside of the data should drive the general form of the model, then the data can help determine the specifics (coefficients, cut-points) of the general model. If one of the general forms is clearly superior, then pretty much any measure should show which is superior (and that should match with the science as well). If you need a specific measure to decide, and different measures are likely to give different results, then a small change in the data would likely make a difference as well. Would you be happy with a data driven decision that would have been different had one single data point not been collected? Knowledge outside of the data will be consistent and better for these choices.

Answer 2

I'm quite surprised that nobody mentioned the existence of other criteria for comparing regression models and selecting the best one. These criteria belong to two different approaches: traditional hypothesis testing and information theory.

The non-mentioned criteria for model comparison and selection, using the former approach, include other error measures (RMSE, MAE, MAPE, MASE, MPE), adjusted R-squared, F-test statistic (also see this). Additionally, to this group IMHO belong goodness-of-fit (GoF) measures, such as chi-squared GoF test statistic and likelihood-ratio GoF test statistic.

As for the latter approach, the non-mentioned criteria include Akaike's information criterion (AIC), Mallows' Cp statistic and Bayesian information criterion (BIC). [NOTE: "Non-mentioned" refers to the time when I started writing this answer, prior to @EngrStudent's answer, which I just saw after posting my answer.]

UPDATE:

Just wanted to add two points. First, another nice measure for regression model comparison and selection is standard error of regression (also referred to as sigma), which is better than R-squared due to maintaining the original data scale. In R parlance, this criterion can be found in lm() output under the name of "residual standard error". Second, for completeness, I would like to mention some non-analytic (exploratory) approaches and criteria to model comparison and selection, such as diagnostic plots (Q-Q, residuals, etc.) as well as domain/theory considerations and model simplicity (parsimony).

Answer 3

I first would suggest a model selection criterion that accounts for model complexity AND residual error. Candidates include Akaike Information Criteria (AIC), Bayes information Criteria (BIC), and many others.

I have used AIC to good effect when selecting among non-parametric models as well (Spline Smoothing). I know this is debating terminology, but non-parametric models have parameters and in many cases have more parameters than the data sample count.

I have not used these models outside their class. While I can pick the best non-parametric model of a family of such models against the data, and I can pick the best parametric (linear, polynomial, analytic, et cetera) model against other parametric models, I do not know if when comparing the AIC of a non-parametric model vs. a parametric model the results will be what I am expecting. This is an interesting line of questioning.

I know that random forests can be used to determine variable importance. Perhaps the Gini score, related to the Kullback-Leibler divergence, is itself an information criteria. If so then one might use a random forest as a model selector or perhaps just a model pruner. Could variable importance in an RF be used to determine whether another RF was a better indicator of variable importance or whether a linear function was.

Perhaps some of these would be relevant:

• http://dl.acm.org/citation.cfm?id=1143865
• http://www.sciencedirect.com/science/article/pii/S0167865510000954

Answer 4

$R^2$ indicated how well your data fits to your model, therefore the higher the $R^2$ value the better. However, if your $R^2==1$ then your model is probably being overfit. For MSE, the smaller it is, the closer the model prediction is to the actual data. So basically, either a high $R^2$ or a low MSE.

See this article on the pitfalls of $R^2$ http://en.wikipedia.org/wiki/Coefficient_of_determination

R2 does not indicate whether:

• the independent variables are a cause of the changes in the dependent variable;
• omitted-variable bias exists;
• the correct regression was used;
• the most appropriate set of independent variables has been chosen;
• there is collinearity present in the data on the explanatory variables;
• the model might be improved by using transformed versions of the existing set of independent variables;
• there are enough data points to make a solid conclusion.

A good overview on how these two quantities are related can be found in What is the difference between "coefficient of determination" and "mean squared error"?

Answer 5

I assume your predicted variable is continuous, not binary.

If a model A is better in $R^2$, a model B is better in MSE, then the model A is uncalibrated. The model B may or may not be uncalibrated.

"Uncalibrated" means that the model, instead of predicting the output variable $Y$, predicts some linear function of it $c_1Y+c_2$.

You can calibrate your model by:

• Divide your training set $T$ to model training set $T_1$ and calibration set $T_2$. Let $T_2$ be, say, randomly selected 20% of your training set.
• Train your model on $T_1$
• Use $T_2$ to obtain $a$ and $b$ such that $Y \sim a\hat y+b$, where $Y$ is an output variable, $\hat y$ is a prediction of your model.
• Train your model on the whole $T$
• From now on, after obtaining any model prediction $\hat y$, transform it as $\hat y_{final} = a\hat y+b$, where $\hat y_{final}$ is the final prediction, $\hat y$ is the raw prediction of your model.

Hope this helps.