Why is a zero-intercept linear regression model predicts better than a model with an intercept?

Question

Many textbooks and papers said that intercept should not be suppressed. Recently, I used a training dataset to build a linear regression model with or without an intercept. I was surprised to find that the model without an intercept predicts better than that with an intercept in terms of rmse in an independent validation dataset. Is the prediction accuracy one of the reasons that I should use zero-intercept models?

Answer 1

I don't think you should choose models simply because they work better in a particular sample, although it is good that you used a training and validation sample.

Rather, look at what the models say about your situation. In some cases a zero-intercept model makes sense. If the DV ought to be 0 when all the IVs are 0, then use a zero-intercept model. Otherwise, don't.

Substantive knowledge should guide statistics, not the other way around

Answer 2

A no intercept model may make sense if two conditions are met. First, there should be a reasonable subject matter knowledge expectation for the intercept to be zero. Second, there should be a reasonable subject matter knowledge expection for the regression line to remain a straight line as you approach zero. Even if both conditions are satisfied, it is wise to run an analysis with an intercept term and verify that the intercept is not significantly different from zero.

(I am assuming that you are talking about a continuous Y and a continuous X.)

Answer 3

This would be understandable if the intercept you obtained was merely noise --not sig. different from zero. (Am I right that the standardized regression coefficients were nearly the same in both models?) If so I don't think you should generalize from this example. When intercepts are sig. and substantial, they add something meaningful to predictive accuracy.

Answer 4

Look carefully at how the rmse or other statistic is computed when comparing no-intercept models to intercept models. Sometimes the assumptions and calculations are different between the 2 models and one may fit worse, but look better because it is being divided by something much larger.

Without a reproducible example it is difficult to tell what may be contributing.

Answer 5

In linear regression, you are fitting:

$y = f(\beta, X) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots $

You fit $\beta$ given training data $(X, Y)$ Suppose you drop the $\beta_0$ and fit the model, will the error in the fit:

$ \sum_i (y_i- f(\beta, X_i) )^2$

be larger than if you included it? In all (non-degenerate) cases you can prove the error will be the same or lower (on the training data) when you include $\beta_0$ since the model is free to use this parameter to reduce the error if it is present and helps, and will set it to zero if it doesn't help. Further, suppose you added a large constant to y (assume your output needed to be $+10000$ than in your original training data), and refit the model, then $\beta_0$ clearly becomes very important.

Perhaps you're referring to regularized models when you say "suppressed". The L1 and L2 regularized, these methods prefer to keep coefficients close to zero (and you should have already mean and variance normalized your $X$ beforehand to make this step a sensible one. In regularization, you then have a choice whether to include the intercept term (should we prefer also to have a small $\beta_0$?). Again, in most cases (all cases?), you're better off not regularizing $\beta_0$, since its unlikely to reduce overfitting and shrinks the space of representable functions (by excluding those with high $\beta_0$) leading to higher error.

Side note: scikit's logistic regression regularizes the intercept by default. Anyone know why: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html ? I don't think its a good idea.