Does multicollinearity affect performance of a classifier?

Question

I know that wikipedia references are sometimes frowned upon here, but this one has me puzzled: Wikipedia - Multicollinearity

I know what multicollinearity is, and today I tried figuring out how/if it would affect performance of machine learning models.

At the beginning of the article, it says

Multicollinearity does not reduce the predictive power or reliability of the model as a whole

...but, as I read on it says that

A principal danger of such data redundancy is that of overfitting in regression analysis models

and I know that overfitting increases variance greatly, and can degrade performance severely.

Are either of these or both of these statements wrong?

Multicollinearity does not increase bias, but it increases variance (overfitting). — William Chiu, Jan 10 '16 at 18:00
So, are both of the above statements still true? And, is there a better way of explaining this? — makansij, Jan 10 '16 at 18:02

score 8 · Accepted Answer · answered Jan 10 '16 at 18:37

8

There is an important qualifier in the continuation of the first cited quote from Wikipedia: "Multicollinearity does not reduce the predictive power or reliability of the model as a whole, at least within the sample data set" (emphasis added).

Unless there is a singular design matrix, multicollinearity does not prevent fitting a model to an individual data sample. I take that to be the point of the first Wikipedia quote. A regression model can capture the data sample well, including all of the peculiarities and noise of the particular data sample, in the presence of multicollinearity.

The problems arise when you try to apply the model outside the original data set to the underlying population. Multicollinearity will severely affect performance outside of the original data sample, as you recognize.

answered Jan 10 '16 at 18:37

EdM

92,183
10
92
267

1

Thank you. So, I suppose by "sample data set" we mean training data set? And then, the fact that the regression model can capture the noise, as you suggest, would mean that the statement cautioning against overfitting has a lot of validity – makansij Jan 11 '16 at 02:32
Any response on my first comment? anyway....I suppose some kind of cross-validation would uncover this problem, because the variance from each training/testing split would cause a lot of MSE. – makansij Aug 26 '16 at 06:30
1

@Candic3 : Yes, the "sample data set" would be the particular sample used for building the model, the training set if you use a training/test set approach. My favorite way to display the problem with multicollinearity is to repeat model building on a few hundred bootstrap samples of the original data. Then see how variable the regression coefficients are among those models, or how poorly those models fit the original data set. Ridge regression can combine information usefully from multicollinear predictors, minimizing overfitting and providing more reliable predictions on new data. – EdM Aug 26 '16 at 13:22

score 0 · Answer 2 · answered Apr 03 '20 at 14:26

Contrary to @EdM's answer, based on the answer here, the performance will not degrade if the test set has the same covariance matrix. In different words, if the correlation between the variables in the test set is the same, the combination of the coefficients and the feature vectors will lead to a valid result. Often this is the case.

Does multicollinearity affect performance of a classifier?

2 Answers2

Linked