4

While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says:

Without regularization, the asymptotic nature of logistic regression i.e (it never quite hit 0 & never quite hit 1) would keep driving loss towards 0 on all examples and never get there, driving the weights for each feature to +infinity or -infinity in high dimensions.

A good example where it holds true is :

Imagine that you assign a unique id to each example, and map each id to its own feature. If you don't specify a regularization function, the model will become completely overfit.

The above in reference to the below loss function: Loss function for Logistic regression

Could anyone help me understand this point via an example or even at intuition level?

Anu
  • 673
  • 7
  • 16
  • 1
    Could you clarify the meaning of "asymptotic"? Presumably this refers to a postulated sequence of datasets, but exactly how do these datasets vary as you go through the sequence? – whuber Nov 05 '18 at 13:35
  • "Asymptotic" means that the regression values never quite hit 0 & never quite hit 1, i.e (0,1) ! according to the explanation I found in the Google Machine learning course referenced above in the web link, it has nothing to with the features space(dataset), it's only with the logistic regression output value.! – Anu Nov 05 '18 at 20:03
  • 2
    I'm afraid that's an insufficient description--nor does it appear to agree with conventional meanings of "asymptotic." It is likely that your source has some kind of model of either collecting more and more data or creating more and more "features" or covariates. It's not possible to answer your question definitively without knowing something about that model. – whuber Nov 05 '18 at 20:06
  • it's a "Linear logistic regression model", Does it help to decipher the confusion? – Anu Nov 05 '18 at 20:14
  • 2
    No, that's not related to the asymptotics. It's a different issue altogether. The intended asymptotics seem to be related to the hypothetical "Imagine that you assign a unique id to each example, and map each id to its own feature." But that's silly--almost any model (whether logistic regression or otherwise) would be overfit because this imagined model supplies a separate parameter for each observation. – whuber Nov 05 '18 at 21:28
  • Thanks, but even I am trying to understand what the author has said in here towards the end if you could suggest something, it would be really helpful to me else, this will remain an open question. :) – Anu Nov 05 '18 at 22:46
  • @whuber this is a description of the problem https://privatebin.net/?2e129e737c0597ea#4mQA6BE4v8mjxv/R2SgCGABCncJO7FvxmWvTU9JiDys= – Heteroskedastic Jim Nov 06 '18 at 01:41
  • I know this is an old question but I got here after searching for the same thing from the Google crash course. After searching a bit there are good answers here to the same question – ru111 Feb 10 '22 at 16:12

0 Answers0