3

I am new to ML and recently came across the L1 and L2 norm.

The tutorials that I read here and here show some circle and diamond plots to explain these topics but I don't really understand the below things

enter image description here

a) Why are we trying to find an intersection point?

b) Does it always have to intersect?

c) I believe, the size of circle and diamond can be larger as well. Right now it's 1 (C = 1) but guess it can be C=2,3 and 4 etc as well

d) I know with linear regression gradient descent, we get the estimate Beta (without any constraints). So, when we use a constraint like Lasso or Ridge, why are we trying to make sure it intersects?

e) I read that this circle or diamond region is called a feasible area. Not sure what that means? Can help, please?

f) I read an explanation Heuristically, for each method, we are looking for the intersection of the red ellipses and the blue region as the objective is to minimize the error function while maintaining the feasibility. May I check what does feasibility mean and why does it have to intersect?

In addition, may I also know in the below graph, why does it have to intersect at the green contour line and not the red contour line

enter image description here

If the contour plot had been placed a little above, we could have got intersection at green instead of red?

what does L1 Norm Isosurface mean?

Can you guys help me?

The Great
  • 3,272
  • 2
    $x_1+x_2=c$ yields a straight, downward-sloping 45° line. Based on that you can see that $|x_1|+|x_2|=c$ yields four lines that form a diamond. $x_1^2+x_2^2=c$ yields a circle (this is the equation for a circle). Probably not very helpful, so just posting this as a comment. – Richard Hardy Feb 09 '21 at 16:05
  • @RichardHardy - Thanks for your help. can help me understand on how do we get the circle plot shown in the question? – The Great Feb 10 '21 at 02:55
  • I understand how gradient descent works. But unable to visualize/think about how this plot came up? – The Great Feb 10 '21 at 03:02
  • @TheGreat Richard answers this in the second sentence of his comment: draw a circle with a radius $c$. From the definition of $L^2$ norm, you can immediately see that $L^2$ norm coincides with the definition of a circle. – Sycorax Feb 10 '21 at 03:18
  • my question is about the multiple circles' thingy and points like 2.000 , 4.00 and 8.00 etc – The Great Feb 10 '21 at 04:28
  • I uderstand those 2,4and 8 are the output (y) based on x1 and x2.. but shouldn't that be just a point...why circle is drawn? – The Great Feb 10 '21 at 04:29
  • It's a circle because multiple values $x_1, x_2$ satisfy $x_1^2 + x_2^2 = 2$, $x_1^2 + x_2^2 = 4$, $x_1^2 + x_2^2 = 8$, etc., and all of those values $x_1, x_2$ form a circle together. – Sycorax Feb 10 '21 at 04:54
  • @Sycorax and Richard Hardy - can help me with the updated post please if you have time? – The Great Feb 10 '21 at 11:42

0 Answers0