8

There are already posts about warnings from R dealing with logistic regression and linear separation such as this one. I just wanna make sure if in Python Scikit Learn this problem is all solved by the L1/L2 regularization part in the optimization function. In other words, it's safe to say users will not get any infinite MLE estimates warnings from sklearn.linear_model.LogisticRegression?

kjetil b halvorsen
  • 77,844
Nicholas
  • 515

1 Answers1

8

Yes, sklearn.linear_model.LogisticRegression uses penalized logistic regression which "solves" the problem of perfect separation. If you set C to something too large, you might still end up with bad results, though.

Danica
  • 24,685
  • Why we need to bother with regularization when data are linearly separable? I mean the weights can go to infinity but the decision boundary won't change. – ado sar Dec 19 '22 at 13:23
  • 1
    @adosar Two things here: computational and statistical. Computationally (what this question was about), it's annoying to have your weights go to infinity; small regularization prevents that. Statistically, remember that if the dimension is greater than the number of data points, then almost any dataset (in the measure theoretic sense) linearly separable; this is an example where you might expect regularization to be important for finding the "right" decision boundary to generalize well. (If the dimension is high but less than $n$, it can still be very helpful, depending.) – Danica Dec 31 '22 at 09:05