Through an L0 and L2 regularization method I have developed a logistic model that has an intercept of around -3. Taking the exp() of that alone I get a probability of around 5%. So that is when other parameters in the model are zero (they cannot be negative).
My question: Is that wrong for a logistic model ? Should they go from 0 to 1 no matter what, and hence my model is completely wrong, or...?
A few thoughts:
The intercept is the log-odds of a positive class membership when all categorical predictors are at their reference levels and continuous predictors are at values of 0. An intercept of -3 represents odds of 0.05 and a probability of 4.7% at those conditions.
Predictor variables can have any reasonable values. The important trick is making sure that their values are linearly related to the log-odds of class membership.
As a probability of 0 means infinitely negative log-odds, a simple logistic regression model will not be able to handle actual probabilities of 0 adequately. It's possible to have perfect separation in a logistic model, with some combination of predictor values exactly distinguishing cases with probabilities of 0 and 1. In that case penalization of the model coefficients is one way to proceed.
Whether you are penalizing to deal with perfect separation or for other reasons, note that penalization seeks deliberately to trade off increased bias for decreased variance. So if you are using a penalized model you should expect some bias.
exp()of that greater than $1$ but this again will be the odds rather than probability - if the intercept had been $+3$ then the odds would be about $20.86$ and the probability about $0.9526$. There is nothing wrong with $5%$ in a particular case if that reflects reality – Henry Jan 19 '20 at 21:48