1

I am comparing the MAE of LASSO regression of multiple features vs. MAE of linear regression of each individual feature, and I am having trouble understanding why the LASSO MAE can be worse than some of the individual feature MAE, even on for the training set (where one single feature resulted in lower MAE than LASSO).

In my understanding, LASSO is a linear regression with regulation to make weight of "un-useful" features zero while minimizing MSE (which should be reflected in minimized MAE as well). Then why did LASSO chose multiple features that gives higher error rather than only keeping a single or fewer features that gives a lower error?

Anna
  • 11
  • Welcome to Cross Validated! Why shouldn't this be possible (or what makes this counterintuitive)? You have fit a model using one loss function and found that the model minimizing it is outperformed by another when it comes to some other loss function. This is not guaranteed, but it happens. – Dave Mar 22 '23 at 11:59

1 Answers1

1

why did LASSO chose multiple features that gives higher error rather than only keeping a single or fewer features that gives a lower error?

You told the regression to minimize the LASSO loss and then evaluated it on a different criterion.

Setting aside numerical issues (LASSO lacks a closed-form solution, after all), minimizing a loss function is literal: such estimation finds the parameters that give the smallest value of that particular loss function. There is no guarantee about another loss function; that would make all loss functions equivalent. It might turn out that the solution giving a smaller value for one loss function also gives the smaller value for another loss function, but minimizing the loss function only guarantees the smallest value for that particular loss function.

Dave
  • 62,186
  • Good point, thank you Dave! This should be obvious but somehow I was overlooking the difference between the lasso loss and MAE! – Anna Mar 22 '23 at 17:54
  • @AnnaXie Depending on what kind of penalty you have in the LASSO loss, you might find a model having lower LASSO loss than another but higher MSE than that same model. It is not just MAE. – Dave Mar 22 '23 at 18:06