3

I'm trying to fit a ridge regression model with a single predictor. However, when I try to do so in three different R packages I get the three following errors:

Error in colMeans(X[, -Inter]) : 
  'x' must be an array of at least two dimensions

Error in if (is.null(np) | (np[2] <= 1)) stop("x should be a matrix with 2 or more columns") : 
  argument is of length zero


Error in colMeans(x[, -Inter]) : 
  'x' must be an array of at least two dimensions

The bottom line from these errors is that x needs to have at least 2 dimensions. Why is this necessary for ridge regression? Does this mean that I can't use ridge regression with a single predictor? Just seems weird I couldn't use ridge regression to get regularization for something like a t-test.

Here is my code:

library(lmridge)
library(glmnet)
library(ridge)

data


set.seet(100)
y <- rnorm(100)
x <- rbinom(100, 1, .5)
z <- rbinom(100, 1, .5)
data <- cbind.data.frame(y, x, z)


ridge


linearRidge(y ~ x, data = data)


glmnet


glmnet(data$x, data$y, nlambda = 25, alpha = 0, family = 'gaussian', lambda = .5)


lmridge


lmridge(y ~ x, data = data, scaling = "sc", K = seq(0, 1, 0.001))
andy_dorsey
  • 105
  • 3
    Out of curiosity, is there any particular reason why you want to use ridge regression? Using ridge regression with a single predictor doesn't carry any benefits I can see, as you are not going to be overfitting with a single predictor. The only time I could ever imagine running into this issue is if I was doing a homework assignment that wasn't fully thought through. – guy Jul 24 '20 at 21:26
  • 2
    There's lot of research showing that experimental intervention effect sizes in the scientific literature tend to be exaggerated. So, if I have a simple two condition experiment, why I not try and regularize the estimated treatment effect towards zero? – andy_dorsey Jul 24 '20 at 22:37
  • How would you choose the penalty size in this specific setting? – Michael M Jul 25 '20 at 07:38
  • ^ No clue. I'm just learning about ridge regression now and I'm definitely not a statistician. – andy_dorsey Jul 25 '20 at 15:58
  • This isn't an answer to the underlying statistical question, but for completeness: in R, when you subset a matrix to get a single row or column, you get back a vector, not another matrix, by default. You need to keep this in mind whenever you have general subsetting that could be a single index. All 3 examples you show fail to do this properly (the two colMeans examples should probably drop = FALSE and the middle example seems to think | short-circuits when it actually does not, so the wrong error message is returned), probably because a single column is not expected behavior. – Chris Haug Apr 20 '22 at 14:00
  • A reason that these functions don't work with vectors instead of matrices could be lazy programming. If you have a matrix X you can refer to the columns of the matrix like X[,1]. But if instead of a matrix you have only a single vector, then X[,1] gives an error. If you would want to improve the function then this means that you need to work with an exception (for the case of only one single variable). – Sextus Empiricus Apr 21 '22 at 22:48

2 Answers2

1

StatQuest does ridge with one predictor just fine in his video.

https://youtube.com/watch?v=Q81RR3yKn30

The method is somewhat silly to use in a regression with just one parameter, but I am surprised the common software implementation don’t allow it. Perhaps the StatQuest example could make sense in some setting.

But that’s just an issue with the software implementation. You’re still able to write your parameter vector as $\hat{\beta}_R = (X^TX+\lambda I)^{-1}X^Ty$ and do the calculation.

($I$ is the identity matrix; $\lambda$ is your ridge regression hyperparameter.)

Another popular software implementation of ridge regression is the sklearn packing in Python. Perhaps give that a whirl.

Dave
  • 62,186
  • 1
    I'm not quite sure why it's silly to use. What if it was for an experiment where the single predictor was a dummy coded variable indicating whether respondents were in the treatment and control. There's lot of research showing that experimental intervention effect sizes in the scientific literature tend to be exaggerated. So why not try and regularize the estimated treatment effect towards zero? – andy_dorsey Jul 24 '20 at 22:36
  • It’s an unusual application of ridge regression, but you have an idea, so run with it and see what happens. I think I have some thoughts on the method, but that would be the subject of a separate question. – Dave Jul 24 '20 at 22:42
  • Ok. Happy to consider any methods that would help ensure that I'm not overestimate the average treatment effect from the experiment. I should also add that I'll also consider covariates in my modeling, so I will have more than one predictor. I'd just like to understand the simpler model with only one predictor first. – andy_dorsey Jul 24 '20 at 22:48
  • The trouble, at least some of the trouble, is that the ridge penalty could squash down the parameter of interest and focus on the covariates. I think it would make for an interesting question! – Dave Jul 24 '20 at 22:56
  • Ridge regression produces a biased estimator. The point of ridge regression is that you reduce the variance in your estimate at the cost of increasing bias (not reducing it as you say you would like). The reason effect sizes in the literature are inflated is because of publication bias (and small sample sizes which produce more variance in effect sizes). Ridge regression solves some problems, but this isn't one of them. You could also make an argument like "I'm going to cut whatever effect size I find in half and report that instead" and that would be similar logic to what you're proposing – Ashish Oct 06 '21 at 19:47
0

As others have noticed, using Ridge regression with only a single feature is overkill. Ridge regression is used when you have many features, so you want to penalize them to avoid overfitting. From your comments, it seems like your idea is to use Ridge regression as a form of prior

There's lot of research showing that experimental intervention effect sizes in the scientific literature tend to be exaggerated. So why not try and regularize the estimated treatment effect towards zero?

As you can learn from the Is Bayesian Ridge Regression another name of Bayesian Linear Regression? thread, Ridge regression is a MAP estimate of a Bayesian linear regression with a Gaussian prior for the parameters. Why not just use a Bayesian model? Ridge regression is a poor man's implementation of such a model, going full Bayesian enables you to be more flexible with model definition and would enable you to define the priors more explicitly. Ridge regression would also somewhat work like this, but it is the wrong tool for the job.

Tim
  • 138,066
  • Bayesian regression is more flexible, but ridge regression has an advantage that it can be computed easily with a matrix calculation. $\hat\beta = (X^TX + \lambda I)^{-1}X^T y$ – Sextus Empiricus Apr 21 '22 at 22:37