6

Being aware of that article, I am curious about the question how big standardized coefficients can get. I had a discussion with my professor about that issue and she was arguing standardized coefficients (beta) in multiple linear regressions can not become greater than |1|. I have also heard that predictors with standardized coefficients greater than 1 should not be be included/appear in multiple linear regression. When I recently estimated a multiple linear regression in R using lm(), I estimated the standardized coefficients with lm.beta() function from the package 'lm.beta'. In the results I could observe a standardized coefficient greater than one. Right now I am just not sure about what is the truth.

Can standardized coefficients become greater than |1|? If yes, what does that mean and should they be excluded from the model? If yes, why?

I would be very thankful, if somebody could make this issue clear for me.

CoderGuy123
  • 437
  • 4
  • 14
Magnus Metz
  • 233

3 Answers3

15

It's never easy telling your professor that they are wrong.

Standardized coefficients can be greater than 1.00, as that article explains and as is easy to demonstrate. Whether they should be excluded depends on why they happened - but probably not.

They are a sign that you have some pretty serious collinearity. One case where they often occur is when you have non-linear effects, such as when $x$ and $x^2$ are included as predictors in a model.

Here's a quick demonstration:

data(cars)
cars$speed2 <- cars$speed^2
cars$speed3 <- cars$speed^3
fit1 <- lm(dist ~ speed, data=cars)
fit2 <- lm(dist ~ speed + cars$speed2, data=cars)
    fit3 <- lm(dist ~ speed + cars$speed2 + speed3, data=cars)
summary(fit1)
summary(fit2)
summary(fit3)
lm.beta(fit1)
lm.beta(fit2)
lm.beta(fit3)

Final bit of output:

> lm.beta(fit3)
   speed    speed2    speed3 
  1.395526 -2.212406  1.681041

Or if you prefer you can standardize the variables first:

 zcars <- as.data.frame(rapply(cars, scale, how="list"))
 fit3 <- lm(dist ~ speed + speed2 + speed3, data=zcars)

 summary(fit3)

Call:
lm(formula = dist ~ speed + speed2 + speed3, data = zcars)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.03496 -0.37258 -0.08659  0.27456  1.73426 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  7.440e-16  8.344e-02   0.000    1.000
speed        1.396e+00  1.396e+00   1.000    0.323
speed2      -2.212e+00  3.163e+00  -0.699    0.488
speed3       1.681e+00  1.853e+00   0.907    0.369

Residual standard error: 0.59 on 46 degrees of freedom
Multiple R-squared:  0.6732,    Adjusted R-squared:  0.6519 
F-statistic: 31.58 on 3 and 46 DF,  p-value: 3.074e-11

You don't need to do it with lm(), you can do it with matrix algebra if you prefer:

Rxx <- cor(cars)[c(1, 3, 4), c(1, 3, 4)]
Rxy <- cor(cars)[2, c(1, 3, 4)]
B <- (ginv(Rxx)) %*% Rxy
B

         [,1]
[1,]  1.395526
[2,] -2.212406
[3,]  1.681041
Jeremy Miles
  • 17,812
  • 2
    Thank you Jeremy for your answer! See a new question here: http://stats.stackexchange.com/questions/120892/standardized-coefficients-for-linear-models-with-numeric-and-factor-variables-in Best regards, Magnus – Magnus Metz Oct 21 '14 at 10:20
5

This is probably a matter of definitions. Does a standardized coefficient refer to standardizing only the predictor variables? or standardizing the response variable as well? I have seen both used to compute "standardized coefficients". Even then, there is more than one way to standardize.

If you divide both the predictor and response variable by their standard deviations (common way to standardize) and fit the regression (with only a single predictor/a single slope coefficient) then it is mathematically impossible to see a coefficient outside of the -1 to 1 range (since the slope will be the same as the correlation). But if you don't standardize the response variable then it would be easy to see an estimated coefficient outside of -1 to 1 depending on the scale of the response variable.

With multiple predictors, an unusually large standardized coefficient could be a sign of multi-colinearity and is probably why some sources suggest dropping those variables.

I expect that the differences between what some sources say are possible and what you observe from others is due to the difference in definitions.

Greg Snow
  • 51,722
  • So am I right, if I conclude that standardized coefficients get values between -1 and 1 only in simple linear regressions, whereas in multiple linear regressions standardized coefficients can become greater than 1, but only, if the response variable is not standardized but the predictors are? I also assume that lm.beta() does not standardize the response variable as I estimated standardized coefficients > 1. Which function in R should be used then to receive standardized coefficients between -1 and 1 as expected from my professor? – Magnus Metz Oct 16 '14 at 21:31
  • 2
    @MagnusMetz, you can standardize the variables yourself. The scale function in R will do this for you. With multiple linear regression and high colinearity among predictors (even standardized) you can still see standardized coefficients outside of -1 to 1 (see Jeremy's answer). You can orthogonalize the predictors (use poly instead of manually creating polynomial terms, use residuals of x2~x1 in place of x2, ...) to overcome the colinearity issues. – Greg Snow Oct 17 '14 at 16:11
  • Greg, the correctness of this answer appears to depend on an implicit assumption that all regressors (apart from a constant) are orthogonal. That is the only way to justify that "the slope will be the same as the correlation." – whuber Dec 17 '16 at 16:24
  • 1
    @whuber, my statement about the slope equaling the correlation is only when there is a single predictor, and by that I meant a single linear term, or transformation, polynomial or spline transforms of a single x-variable would be multiple predictors as far as the computations go. Once there are multiple predictors then the standardized slope will not necessarily equal the correlation. Thanks for seeing the possible confusion and helping clarify. – Greg Snow Dec 19 '16 at 15:56
  • @Greg Since you have indicated that subtle point only indirectly (by referring to "the" predictor), I would only like to suggest it would help to include it explicitly in your answer itself. – whuber Dec 19 '16 at 17:14
  • @whuber, does the edit I made make it clear enough? or would you suggest something more? – Greg Snow Dec 19 '16 at 22:49
  • It works for me. +1 – whuber Dec 19 '16 at 22:52
1

A standardized beta weight greater than one is a sign of suppression, especially cooperative suppression. Such suppression increases the predictive value of the predictors and thus is of potentially great value. See http://core.ecu.edu/psyc/wuenschk/MV/multReg/Suppress.docx

Karl L. Wuensch
  • 11