3

In the elements of statistical learning book the lasso is written like this:

$\hat{\beta}^{lasso}=\operatorname{argmin}\left \{ \frac{1}{2}\sum_{i=1}^{N}(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j)^2 +\lambda \sum_{j=1}^{p}|\beta_j| \right \}$

Note that before the sum 1/2 is used. In some cases I see it as 1/(2N). Like e.g. this equation:

$\hat{\beta}^{lasso}=\operatorname{argmin}\left \{ \frac{1}{2N}\sum_{i=1}^{N}(y_i-\sum_{j=1}^{p}x_{ij}\beta_j)^2 +\lambda \sum_{j=1}^{p}|\beta_j| \right \}$

I don't understand what this would represent if I divide with the number of observations? I would very much appreciate any clarification.

Edgar
  • 1,651
  • 2
  • 10
  • 28
Ville
  • 857

1 Answers1

6

It doesn't really matter from an algorithmic or solution perspective, because $\lambda$ is a "free variable" that is typically selected by cross-validation or some such. If you divide the first term in your first equation by $N$, and decrease the value of $\lambda$ by a factor of $N$, you'll have exactly the same solution.

People sometimes divide the first term by $N$ because that gives you the "Mean Squared Error" (well, $\text{MSE}/2$ in this case) which is a commonly-used metric for evaluating accuracy. If you don't divide by $N$ you get the "Sum of Squared Errors" (SSE), which varies with your sample size, which in turn makes it more difficult to compare, for example, accuracy for various sub-populations of the data that may be of different sizes.

The reason for dividing by $2$ is that when you take the derivative, the $1/2$ cancels out the "$2$" created by the exponent of the squared error term, making subsequent typing of the math slightly shorter and saving a small amount of mental effort remembering why that $2$ (that would be there if not for the cancellation) is in the subsequent equations. Again, it's irrelevant to the actual solution, as $\lambda$ can just be rescaled to match.

jbowman
  • 38,614