2

It is well known that $\beta = (X^{T}X)^{-1}X^{T}Y$ for linear regression. While experimenting with this, I found that $\beta$ (excluding the intercept term) remained the same before and after demeaning $X$. The p-values for coefficients' t-tests also remained the same.

From a statistical sufficiency perspective, a well-known property is that the sample mean is independent of sample variance under normality. This can potentially explain the unchanged p-values. But I am struggling to see the connection to unchanged $\beta$.

Can anyone provide an intuition or proof for this?

Kaiwen Wang
  • 51

2 Answers2

6

Intuition is as follows: Without demeaning, $\hat{\beta}$ is the MLE for $$Y_i = \beta_0 + x_{i 1}\beta_1 + \dots + x_{i p}\beta_p + \varepsilon_i,$$ assuming $\varepsilon_1, \dots, \varepsilon_n$ are I.I.D. $N(0, \sigma^2)$. With demeaning, $\hat{\alpha}$ is the MLE for $$Y_{i} = \alpha_0 + (x_{i 1} - \bar{x_1})\alpha_1 + \dots (x_{i p} - \bar{x_p})\alpha_p + \varepsilon_i = \alpha_0 - \bar{x_1}\alpha_1 - \dots - \bar{x_p}\alpha_p + x_{i 1}\alpha_1 + \dots + x_{i p}\alpha_p + \varepsilon_i.$$ Clearly these are just two different parameterizations of the same model, as the relationship between the parameters in the two parameterizations is $$\beta_0 = \alpha_0 - \bar{x_1}\alpha_1 - \dots - \bar{x_p}\alpha_p,$$ $$\beta_i = \alpha_i \text{ for } 1 \leq i \leq p.$$ Therefore the relationship between MLEs is the same. Test $p$-values will also be the same as long as you are keeping the reparameterization in mind.

Kakashi
  • 161
3

Here's something more ...intuitional, if that's a word.

The $\beta_i$ coefficients are the predicted increase in $Y$ for each unit increase in $X_i$ and the increase is linear.

Removing the mean (or any other value) from $X_i$ just moves along that straight line.

Peter Flom
  • 119,535
  • 36
  • 175
  • 383
  • 6
    “Intuitive,” perhaps? – Sycorax Nov 10 '23 at 21:48
  • 1
    Another way of looking at this is to observe that simple linear regression goes through the point $(\bar x, \bar y)$ and this continues to be the case if you relocate all the data; if that relocation is subtracting the means, it will now go through $(0,0)$ – Henry Nov 11 '23 at 10:08