1

I am taking a course about ridge, bridge and lasso regression. My teacher showed us that the solution to ridge regression can be written in the form

$$ \beta=(X^TX+\lambda I)^{-1}X^T Y $$

My question is do lasso and bridge regression have a similar solution?

user3741635
  • 148
  • At least for LASSO this has been answered already in the site (albeit it's written in the question). – Firebug Nov 01 '17 at 15:39
  • @Firebug, I have already checked that question before asking my question but I didn't understand what closed form lasso is and how it is related to lasso regression – user3741635 Nov 01 '17 at 15:53
  • It's written right at the beginning: $$\beta^{\text{lasso}}= \operatorname*{argmin}_\beta | y-X\beta|^2_2 + \alpha | \beta|_1$$ – Firebug Nov 01 '17 at 16:04
  • At the end of the question is a demonstration why LASSO regression has no closed form solution (in fact some exist for special cases, like orthogonal dependent variables). – Firebug Nov 01 '17 at 16:05
  • @Firebug That link doesn't demonstrate that lasso has no closed form solution: it just shows that the derivation attempted by OP is wrong. (I edited the title of the post last month since I saw someone else mistakenly reference it in the same way.) – user795305 Nov 01 '17 at 22:10
  • @user3741635 The ridge regression estimate is defined as $\beta_\mathrm{ridge} = \arg\min_\beta\frac{1}{2}|y-X \beta|2^2+\lambda|\beta|_2^2$. Your teacher showed that it isn't necessary to perform a numerical minimization to compute $\beta\mathrm{ridge}$ by finding a closed form $\beta_\mathrm{ridge} = \left(X^TX+\lambda I\right)^{-1}X^Ty$. Bridge and lasso are defined similarly to ridge, except with the "$2$"s in the last term replaced with $\gamma \in [1,2)$. However, for those problems, there does not exist a closed form, so numerical minimization must be performed. – user795305 Nov 01 '17 at 22:16
  • @Ben The question does in fact show LASSO has no closed form:

    "We can see now that our solution for one $\beta_j$ is dependent upon all the other $\beta{i\neq j}$ so there is no closed form solution."_

     – Firebug Nov 01 '17 at 22:20
  • @Firebug $x = 2y$ and $y = 2 + 3x$ has each term depending on the other terms. Getting stuck on one approach doesn't mean no closed form exists. – user795305 Nov 01 '17 at 22:22

0 Answers0