1

I am studying some properties of the least squares estimator and I have the following statement, with $X$ a full rank matrix

If $e$ is independent of $\hat y = X \hat \beta$, then $S^2=e^Te/(n-p)$ will be independent of $\hat \beta$.

The teacher quickly said that $\hat y$ is one-to-one with $\hat \beta$ since $X$ is full rank, so if $e$ is independent of $\hat y = X \hat \beta$ then it is also independent from $\hat \beta$ then $S^2$ is independent of $\hat \beta$

I don't understand why is this true and I feel this reasoning is not very rigorous. Can someone make it clear ?

Kilkik
  • 345
  • Related: https://stats.stackexchange.com/questions/117406/proof-that-the-coefficients-in-an-ols-model-follow-a-t-distribution-with-n-k-d?noredirect=1&lq=1 https://stats.stackexchange.com/questions/20227/why-is-rss-distributed-chi-square-times-n-p?rq=1= – Christoph Hanck Oct 24 '22 at 07:26

1 Answers1

3

$S^2$ is independent of $\hat{\boldsymbol\beta}.$

To see that, notice $\mathbf y\sim \mathcal N_n(\mathbf X\boldsymbol\beta, \sigma^2\mathbf I_n),~S^2=\frac{(\mathbf y-\mathbf P\mathbf y) ^\mathsf T(\mathbf y-\mathbf P\mathbf y)}{(n-r) }$ and

\begin{align}\operatorname{Cov}\left[\hat{\boldsymbol\beta},\mathbf y-\mathbf X\hat{\boldsymbol\beta}\right]&= \operatorname{Cov} \left[\left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\mathbf y, (\mathbf I_n-\mathbf P) \mathbf y\right]\\&= \left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\operatorname{Cov}[\mathbf y,\mathbf y](\mathbf I_n-\mathbf P) ^\mathsf T\\&= \sigma^2\left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T(\mathbf I_n-\mathbf P)\\&=\mathbf 0.\tag 1\end{align}

Reference:

$[\rm I]$ Linear Regression Analysis, George A. F. Seber, Alan J. Lee, John Wiley & Sons., $2003, $ Theorem $2.5, $ p. $25;$ Theorem $3.5,$ p. $48.$

User1865345
  • 8,202