1

Reading the Scikit-learn docs on logistic regression (https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression), I came across this note:

Note: Regularization is applied by default, which is common in machine learning but not in statistics.

Would it be right to say that the idea of regularization (especially the ridge l2 or lasso l1 penalty) is a contribution that came out of the machine learning community? Or were such methods already known to statisticians?

For instance, ridge regression and lasso are mentioned in Elements of Statistical Learning, but I am unsure whether the authors are representative of most statisticians, or if they should rather be considered machine learning researchers. Thank you!

Enk9456
  • 33
  • 1
    Ridge regression was introduced by Hoerl & Kennard in 1970, so well before the ML/AI vs. statistics debate. – chl Jan 04 '21 at 20:24
  • The reason for that comment in sklearn is that when most of the world talks about logistic regression (statisticians, scientists etc) it is performed without regularisation, since essentially you cannot perform inferential statistics with regularised models. therefore they are "apologising" for the fact that they are applying regularisation. in particular, whilst many other ML libraries have a default of 0 regularisation, their regularisation parameter is 1/(regularisation strength), so one would have to specify infinity. given this they chose an arbitrary regularisation of 1 as default. – seanv507 Jan 04 '21 at 21:12

1 Answers1

1

Ridge regression was introduced before machine learning existed as a field. In the statistics literature, the ridge estimator is typically attributed to Hoerl and Kennard (1970). Even before that, essentially the same method was introduced in different contexts by Andrey Tikhonov and independently by David Phillips. In machine learning literature ridge regression was reinvented via the name of weight decay.

In the statistics literature, the lasso was introduced by Robert Tibshirani (1996). A closely related technique (called basis pursuit) was independently introduced in signal processing literature at roughly the same time by Chen, Donoho and Saunders (1994, 1998).

user13243542345452
  • 405