11

What are the disadvantages of using elastic net in comparison to lasso. I know that the elastic net is able to select groups of variables when they are highly correlated.

  1. It doesn't have the problem of selecting more than $n$ predictors when $p \gg n$. Whereas lasso saturates when $p \gg n$.

  2. When there are highly correlated predictors lasso tends to just pick one predictor out of the group.

  3. When $n \gg p$ and the predictors are correlated, the prediction performance of lasso is smaller than that of ridge.

All these disadvantages of lasso are overcome by the elasic net.

What I don't understand is when should the lasso be then used? Is there any reason to use it when the elastic net performs better than lasso? What are the disadvantages of using elastic net in some cases? In which cases would the lasso be a better choice?

Richard Hardy
  • 67,272
Ville
  • 857

2 Answers2

15

One disadvantage is the computational cost. You need to cross-validate the relative weight of L1 vs. L2 penalty, $\alpha$, and that increases the computational cost by the number of values in the $\alpha$ grid.

Another disadvantage (but at the same time an advantage) is the flexibility of the estimator. With greater flexibility comes increased probability of overfitting. It may be that the optimal $\alpha$ for the population and for the given sample size is $0$, turning elastic net into lasso, but you happen to choose a different value due to chance (because that value delivers better performance when cross-validating in the particular sample).

Richard Hardy
  • 67,272
  • 1
    Minor comment with regards to your second point: lasso and elastic net are estimators for the same model. As such, there is no difference in flexibility of the model. – hejseb May 09 '18 at 22:15
  • 2
    @hejseb: With LASSO, there is a single parameter that is optimized over during cross-validation: $\lambda$. In elastic-net, you can optimize over both $\alpha$ and $\lambda$, meaning more opportunity for overfitting during the cross-validation selection process. On the other hand, just using the default values of $\alpha$ tend to perform really well, so often only $\lambda$ is optimized over. So I disagree with your statement, because I consider $\alpha$ and $\lambda$ a part of the model (although I understand the ambiguity). – Cliff AB May 09 '18 at 22:33
  • 3
    @hejseb, excellent point! Now corrected. Cliff AB, I think the model definition need not include the tuning parameters of the estimator (lasso, elastic net, ...) – which $\alpha$ and $\lambda$ are – so to me hejseb's comment makes a lot of sense. I think along the lines of defining a model for the population (a linear model in this case) and estimating its parameters (which include $\beta$s but not $\alpha$ or $\lambda$) by some estimator. – Richard Hardy May 10 '18 at 07:04
0

A very late reply, but if you're interested in tuning the elastic net: my blog post explains why it is actually very hard to tune alpha and lambda together, which is a disadvantage.

Here's the argumentation in a nutshell.

First empirical. When one plots cross-validated likelihood (CVL) against alpha and lambda one will notice a ridge in the landscape along which the CVL is very flat and close to the maximum. CVL is strongly linked to marginal likelihood, which is another criterion to tune hyperparameters (= empirical Bayes). For marginal likelihood one can prove that for elastic net this is approximately a Gaussian likelihood with only one variance parameter v. An infinite number of alpha-lambda combinations can render v, implying non-identifiability.

Mark van de Wiel
  • 91
  • 3
    Welcome to [stats.se]! Please note that you must explicitly state you're linking to your own blog post, see the Help Center. – Glorfindel Sep 06 '22 at 08:55
  • 5
    I am sorry for the harsh response to your posts, Mark. I find your blog post interesting, and so believe the downvoting reflects community reactions to the appearance of abusing site policies and norms. It's worth a few minutes to skim our [help] so you can avoid such problems. I hope you will be inspired to contribute to CV again. – whuber Sep 06 '22 at 11:53
  • 1
    Thanks, Wolfgang. Will try to behave better in the future. – Mark van de Wiel Sep 06 '22 at 19:14