2

I was solving one quiz question on Coursera and I found an interesting question.

If you double the value of a given feature (i.e. a specific column of the feature matrix), what happens to the least-squares estimated coefficients for every other feature? (assume you have no other feature that depends on the doubled feature i.e. no interaction terms).

My questions are:

  1. I think, the other coefficients will stay constant. If so, can someone tell me the logical explanation behind it?
  2. What about the coefficient of the scaled feature? Will it be equal to $m/2$ assuming we doubled the feature and existing weight was $m$?
  3. What if interaction terms are included as well? What will happen there?

So, collectively I can say that regression coefficients are independent of change of origin but not of scale. Will this be correct? If we shift, the coefficients remain unchanged. However will the coefficients change when we scale them?

Shawn Hemelstrand
  • 13,543
teddcp
  • 155
  • 2
    Related: https://stats.stackexchange.com/questions/208341/scaling-in-linear-regression/208344#208344 – Christoph Hanck Jan 01 '24 at 20:14
  • 1
    @ChristophHanck Good find. That answers the first two questions. Only the third is a new one. But all three are thoroughly analyzed and answered in the duplicate ;-). – whuber Jan 01 '24 at 20:20

1 Answers1

3

We can simulate all of this in R and see what happens. Here I simulate three normally distributed variables with slope of $\beta = 2$, with some random noise added in to simulate random error. Then I simply multiply the first predictor by $2$ and refit the regression:

#### Sim Data ####
set.seed(123)
x1 <- rnorm(100)
x2 <- rnorm(100)
y <- (2*x1) + (2*x2) + rnorm(100)

Normal Fit


fit1 <- lm(y ~ x1 + x2)


Refit with Constant


x.multi <- x1 * 2
fit2 <- lm(y ~ x.multi + x2)

We can run coef(fit1) and coef(fit2) to check the coefficients:

 coef(fit1)
(Intercept)          x1          x2 
  0.1350654   1.8668285   2.0238113 
> coef(fit2)
(Intercept)     x.multi          x2 
  0.1350654   0.9334142   2.0238113

We can see that because multiple regression already estimates the slope as the increase in $y$ when all other inputs are $0$, this does not affect the other slopes. However, we have applied a simple linear transformation which is a constant, thus the slope is now halved (because the scale of the variable changed, which is visualized further below). We can check this directly by simply multiplying the refit slope by $2$:

> coef(fit2)[2] * 2
 x.multi 
1.866828

Because an interaction is simply the product of the two predictor terms, we should get no change in the main effect of the covariate, but otherwise changes in both the main effect of our multiplied predictor and the interaction. We fit both forms below and check their coefficients:

#### Refit with Interaction ####
fit.int.1 <- lm(y ~ x1 * x2)
fit.int.2 <- lm(y ~ x.multi * x2)
coef(fit.int.1)
coef(fit.int.2)

We can see this indeed happens, as it once more multiplies the main effect by $2$, but also does the same with the interaction:

> coef(fit.int.1)
(Intercept)          x1          x2       x1:x2 
  0.1409828   1.9071923   2.0343435   0.1591073 
> coef(fit.int.2)
(Intercept)     x.multi          x2  x.multi:x2 
 0.14098276  0.95359616  2.03434352  0.07955363

You'll notice this isn't really doing much to change the actual magnitude of the relationship between $x_1$ and $y$, only the scale of $x_1$ (see the $x$ axis in both plots to see why). The adjusted $R^2$ in each case is approximately $0.8759$:

par(mfrow=c(1,2))
plot(x1,y)
abline(fit1,col="red")
plot(x.multi,y)
abline(fit2,col="red")

enter image description here

Shawn Hemelstrand
  • 13,543