Adding up regression coefficients in the presence of multicollinearity and reversal of signs?

Question

My time series model suffers from multicollinearity between two independent variables.

When taking one of the variables out of the model I obtain a coefficient of 0.08 for variable 1 but when including both variables simultaneously the sign reverses and I get a coefficient of -0.33 for variable 1 and 0.45 for variable 2.

My professor said that if both variables were perfectly correlated the sum of both coefficients would be 0.12 and that 0.08 was therefore a sign for multicollinearity. Does anyone know a paper confirming what she said?

Also when running the regression exluding variable 1, I was expecting the coefficient for variable 2 to be somewhat similar to the coefficient of variable 1 (using the same logic as above, i.e. adding up the coefficients should yield a coefficient that is similar to running the regression alone) but the coefficient is very different (0.3). Can somepne explain why this might be the case?

What I miss to hear from you, @Wiebi, is whether 1) your variables (those referred to above) are of different lags, 2) whether they are of primary interest to the research question and 3) whether your model includes other variables than these two variables. — Daniel, Jul 21 '15 at 17:47
You will probably have to open your statistical textbook for a decent answer. However, a starting point would be to consider the sign of the correlation coefficient or covariance (you can use the correlation coefficient as well as the standard deviation for each of the two variables to compute the covariance). This is because the beta of any variable in OLS, in part, is determined by its variance-covariance matrix, and because your model includes only two variables, it should be relatively easy to grasp why you observe the results you observe. — Daniel, Jul 21 '15 at 18:29
With limited information about your model, your data in general, and so forth, that is probably the best answer I can give you. — Daniel, Jul 21 '15 at 18:30
@Daniel I agree with your statement about generality. But based on the original question by Wiebi I believe my answer and subsequent comments are entirely correct. Also, for the record I think this does not qualify as an answer, but rather clarifying comments. It should be removed and the relevant pieces appended as comments to my answer (first half) and the original question (second half) for the sake of clarity for future readers. — Gavin M. Jones, Jul 21 '15 at 19:28
@GavinM.Jones, I didn't recognise that you had added another comment (i.e. clarified to Wiebi that your conclusions hold only when you are in a 2-variable scenario) There you go :) Hope its okay now — Daniel, Jul 21 '15 at 19:38
This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post. — gung - Reinstate Monica, Jul 21 '15 at 20:23

score 1 · Answer 1 · answered Jul 21 '15 at 12:39

Why don't you check the variance inflation factors, if you are worried about multicollinearity? To me, your results indeed sound like multicollinearity, but there is no way to check for this by eyeballing, i.e. you cannot simply state that multicollinearity is an issue due to the behaviour of coefficients or standard deviations. Neither will your correlation coefficients as I don't know any person who agree what an acceptable correlation coefficient should be. You must check the variance inflation factors to judge whether multicollinearity is a serious issue in your model. Any decent statistical software has an option of inspection the variance inflation factors. Typically, these need to be below 5, at least for your interest variable. Otherwise, multicollinearity could be problematic. If you find that multicollinearity is, indeed, present in your model, then check the Wikipedia-page for Multicollinearity, in particular the "Remedies for multicollinearity"-section. There are many strategies for dealing with multicollinearity, once you have realised that multicollinearity is a problem in your statistical model.

score 1 · Answer 2 · edited Apr 13 '17 at 12:44

Strange behavior in coefficients and SE can be quite indicative of problems with multicollinearity, particularly when sign-switching of coefficients occurs upon adding new predictor variables (see here for a similar discussion).

If you think about it, we are actually fitting a plane through the cloud of data points when we have 2 predictors. If $x_1$ and $x_2$ are highly collinear, then the plane is fitting to a cloud of points that resembles a line or a tube instead of a dispersed cloud (as would be the case with less correlated variables). This results in instability and sign-switching!

I'm not sure about what your professor said regarding summing coefficients to as a sign of collinearity. However, I see three ways to diagnose multicollinearity in this case (two "art", one "science"):

Plotting $x_1$ ~ $x_2$ and checking correlation coefficients. This should be your first step in exploring the relationship among predictors. (Although as another user alluded to, there is no hard cutoff for what is acceptable; this varies by field.)
Sign-switching of coefficients when a new predictor is added, or large changes in SE. This is evidence of instability in the model which could be caused by collinearity. This might lead you to #3.
Look at variance inflation factor (VIF) for a more quantitative measure of multicollinearity. (Although there is still some "art" to deciding where the cutoff is.)

In fact, the bivariate correlation coeffcients are high (> 0.8) but VIFs are low (<5) - even when dividing the sample into subperiods. That is kind of why I am trying to look for other ways to think about the problem and how to intepret it. — Wiebi, Jul 21 '15 at 15:17
@Wiebi it is important as you go along to remember that there is an exponential relationship between VIF and and the correlation coefficient (r) see here:
VIF = 1/(1 - r^2).

Therefore (to the best of my knowledge) the correlation coefficient cannot be "high" but the VIF "low" at the same time; they are congruent. For example, a VIF of 10 indicates that r^2 = 0.9. A VIF of 5 indicates that r^2 = 0.8. — Gavin M. Jones, Jul 21 '15 at 17:07
When you are working with only 2 variables, the VIFs will be equal and the above comment will be true. — Gavin M. Jones, Jul 21 '15 at 18:09

Adding up regression coefficients in the presence of multicollinearity and reversal of signs?

2 Answers2