0

When p>n, number of features greater than the number of observations, LASSO selects only n variables before saturates.

What is the reason for this? Although I have read that the reason is the nature of the convex optimization problem, I still can not understand why?

Bear in mind that I am new to this subject.

Peter Smith
  • 319
  • 2
    Have you considered one of the simpler situations such as $n=1,p=2$? That ought to make the reasons perfectly clear. – whuber Nov 10 '18 at 13:01
  • @whuber, sorry, I even do not understand that, may be because I do not understand “the nature of the convex optimization problem”. So,could you please clarify you comments in details? – Peter Smith Nov 10 '18 at 13:13
  • 1
    Invent data. Write the model. Solve it. You don't need to know a thing about convex optimization, Lasso, or even statistics--that's why this is such an instructive approach. – whuber Nov 10 '18 at 13:19
  • @whuber, I am new to this field :( – Peter Smith Nov 10 '18 at 13:33
  • 1
    Whuber is trying to get you to develop your understanding. Write out a concrete example on paper ( if you get stuck, add example to your question). Try to solve the equations manually. ( Throw convex optimization out of your head)!!! – seanv507 Nov 10 '18 at 16:04
  • @seanv507, but I do not know what to write to develop my understanding. The step by step explanation will allow me to understand the idea. – Peter Smith Nov 10 '18 at 20:37
  • @whuber always like the way you encourage people to think more. – Haitao Du Nov 11 '18 at 10:46
  • 1
    @PeterSmith here is an example n=1, p=2 x = [1,2] y = 5. can you come up 5 solutions to get y = w_1 * x_1 + w_2 * x_2 (ie w_1,w_2), including solution using only x1 or only x2? ? what is the l1 norm of the solutions? – seanv507 Nov 16 '18 at 15:19
  • @seanv507 Thanks for your answer, but I do not completely understand, I understand the equation of the straight line and, 2 parameters, one dependent variable, and two independent variables. – Peter Smith Nov 18 '18 at 11:56
  • 1
    so 1) what are some solutions for w_1 and w_2? [forget the optimum L1 norm for now] 2) what is the L1 norm of the corresponding solutions = |w_1| + |w_2| ? – seanv507 Nov 18 '18 at 15:46
  • @seanv507 1) w_1= (y-w_2x_2)/x_1, and w_2=(y-w_1x_1)/x_2. 2) I do not know. – Peter Smith Nov 18 '18 at 20:24
  • So now fill in the values for x1,X2 Y as in my example. What are possible values for w1 and w2 – seanv507 Nov 18 '18 at 21:34
  • @seanv507, w_1=5-2*w_2 and w_2=(5-w_1)/2 – Peter Smith Nov 19 '18 at 08:11
  • 1
    so pick an number for w_2 and solve for w_1. then calculate the norms for the solution. in particular try w_2 =0, try w_1 =0 (and solve for w_2) and try a few things in between. how many solutions are possible? – seanv507 Nov 19 '18 at 11:49
  • @seanv507, if choose w_2=1 that mean w_1=5, if w_1= then w_2=2. When w_1=0 then w_2=2.5. Also if w_2=0 then w_1=5. So the how many solutions depend on how many values for w. If The norm is an absolute value, it will retain the positive number. – Peter Smith Nov 19 '18 at 12:01
  • Let us continue this discussion in chat. – seanv507 Nov 19 '18 at 12:04
  • @seanv507, I have a problem I can not enter the discussion. The smaller the values of w the smaller the l1 norm – Peter Smith Nov 19 '18 at 13:04

0 Answers0