2

NOT A DUPLICATE For the persons who marked this question as a duplicate of the post I mentioned in my original post: this is not a duplicate, as the correlation obtained with an intercept-only linear model would be NaN or 0, not 0.1 as mentioned in my post. Further I am asking how to use the most common R squared formulation and no answer is provided to that post.

Original post

I used caret with glmnet and selected repeatedcv (10-fold, 5 repeats) to choose the best glmnet parameters (alpha and lambda) in terms of Rsquared (i.e. better = larger Rsquared).

library(glmnet)
library(caret)
alpha.grid <- (1:20) * 0.05
lambda.grid <- 10^seq(4,-4,length=200)
EN.param.grid <- expand.grid(.alpha=alpha.grid, .lambda=lambda.grid)
train.params <- trainControl(method="repeatedcv", number=10, repeats=5)
EN.fit <- train(x=X, y=Y, method="glmnet", tuneGrid=EN.param.grid, 
                trControl=train.params, standardize=TRUE, metric="Rsquared")

Despite the best model having no remaining predictors (i.e. only an intercept), the associated Rsquared as provided by caret (as seen in EN.fit$results$Rsquared dataframe) should be 0 but it is not (in my case 0.1). Then, questions coming to mind are:

  1. Is the test Rsquared calculated in caret in terms of correlation instead of the more traditional sum-of-squares method (as suggested by this post)?
  2. How to correct this behavior to obtain a test Rsquared of 0 for an intercept-only model (as many would expect), and still be able to correctly optimize a glmnet parameter search on Rsquared?
michael
  • 303
  • This remains a duplicate. The Rsquared calculation used by caret is explained in the linked thread. The question of whether the calculation in the ("suggested by") the linked thread is right is nonsensical. 1) If that's what this community has put forward in the past, why do you think this community will proffer a different answer now? 2) The answer shows the actual code that is being used, so it's pretty canonical. 3) The answer is from the author of caret. Your second question about how to get R / caret to provide a different Rsquared is off topic here. – gung - Reinstate Monica Jul 19 '17 at 18:48
  • Dear @gung, I am not asking if the calculation is right (how did you come to that conclusion?). Further 1) I am not asking the same question, but how the test R squared I obtained is different from 0 as 2) the actual code in the "duplicate" would not produce a value of 0.1 as the correlation between a vector of predicted values and a constant would be NaN (because no standard deviation can be calculated) or 0. 3) I do not see how a R squared based on sum of squares would be off topic as it implemented in the code mentioned in that post, and there may be a way to call that part – michael Jul 19 '17 at 19:04
  • The reason I think you are asking if the other answer is right is because your question 1 reads, "Is the test Rsquared calculated... as suggested by this post)?" The answer is clearly yes. Beyond that, this question appears to be off topic. – gung - Reinstate Monica Jul 19 '17 at 19:28
  • Dear @gung, I understand what you are saying but could you please read more carefully my answer? The answer is NOT clearly yes because the code in that post would provide NaN and NOT 0.1. You can see it yourself by typing cor(rnorm(10), rep(42,10))^2 in R (an example with 10 values and an intercept-only model with value 42). – michael Jul 19 '17 at 19:31
  • I am aware of that (cf, here). It would appear from the linked thread that the traditional version is used in that case. – gung - Reinstate Monica Jul 19 '17 at 19:39

0 Answers0