1

I am trying to understand logistic regression, and I have a very basic question. Given that the sigmoid function $\phi(z) = \frac{1}{1+exp(-z)}$ gives 0 or 1 only when $z=-\inf$ or $z=\inf$ respectively, I think it is impossible that we get exact 0 or 1 fitted (or predicted) probability for any combination of the finite number of real-valued input variables. Is that correct?

amoeba
  • 104,745
user5054
  • 1,549
  • 3
    Yes. Though real number arithmetic, and the floating point arithmetic used in all computerized computations are different, so it is possible to get a zero or one to within the precision that a computer can discern. – Matthew Drury May 01 '17 at 03:46
  • Why is this a concern to you? How regression model could possibly know that something is sure to happen, or absolutely impossible? – Tim May 01 '17 at 04:17
  • I have a related question, is there an alternative to the sigmoid function that is used that does allow 1 or 0 to be fitted? – Chogg May 01 '17 at 05:04
  • 1
    There are alternatives but (afaik) they wouldn't have the same nice properties, e.g. they wouldn't be smoothly differentiable or monotonic. For example you could have a "double-rectified" linear function $f(x;a,b)=\operatorname{min}(1, \operatorname{max}(0, ax+b))$, but the disadvantage of that is it isn't differentiable at 0 or 1, and is susceptible to plateaus during learning since the gradient is flat when $ax+b<0$ or $ax+b>1$. Also probabilities of 1 or 0 are generally best avoided unless you're actually dealing with logical certainty or impossibility. – Ruben van Bergen May 09 '17 at 10:09
  • This occurs often with separation, see https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression – kjetil b halvorsen Sep 01 '22 at 22:49

0 Answers0