2

There is a nice answer, however it goes from another way around: the model gets more bias if we drop some features by setting the coefficients to zero. Thus, overfitting is not happening.

I am interested more in my large coefficients indicate the overfitting. Lets say all our coefficients are large. My intuition is that the larger coefficients get the less important are the features. However, if this is the case how are we able fit too much to the data if features are less important?

Alina
  • 1,165
  • 3
  • 11
  • 22
  • 1
    Could you explain that intuition? It would seem to many people, on the contrary, that large coefficients correspond to features that have strong relationships with the response. Please note that we must understand the sense of "large" by assuming all variables have been standardized; otherwise, the size of a coefficient has no inherent meaning. – whuber May 18 '17 at 22:09
  • 1
    I was thinking if we change the units of out features, the coefficients might get larger (or smaller) but the predictions will be unaffected. So, large coefficients does not mean that features are important. However, if we standardize the features a large coefficient can mean that the feature is important, can't it? – Alina May 18 '17 at 22:22
  • 4
    Changing the units of the features does not change the standardized features at all, and therefore is irrelevant. With standardized features, a larger coefficient means a larger change in the response is associated with a unit change in the feature on the scale used to standardize the feature (usually a standard deviation scale). – whuber May 18 '17 at 22:29
  • Are you thinking of situations where collinearity causes two coefficients to get very large but of opposite signs? – Wayne May 18 '17 at 23:40
  • @whuber lets say we standardized our features and got large coefficients. why is it a sign of overfitting? Is it because by changing a feature by one unit there is a large change in the response, meaning with this coefficient we artificially induce the exact fir to each point? – Alina May 19 '17 at 06:57
  • Illustration: if thermonuclear weapons were rated in milligrams of TNT rather than megatons of TNT, these units would not affect any explosion effects modeling - but the units being different, the magnitude of the model parameters would be different. – James Phillips May 19 '17 at 11:56
  • @JamesPhillips yes but how that does it imply that large coefficients mean overfitting? – Alina May 19 '17 at 16:50
  • My comment begins with "Illustration:" to show that large coefficients by themselves are not proof of overfitting. – James Phillips May 20 '17 at 13:28

0 Answers0