2

Coefficients $\beta$ can be estimated from $y$ by weighted least squares with:

$ \hat\beta = (X^T\Sigma^{-1}X)^{-1} X^T \Sigma^{-1} y $

where $\Sigma$ is the covariance matrix of the noise.

Let $N$ be an orthonormal basis of the null space of the design matrix $X$. So the columns of $N$ are orthogonal to the columns of $X$, and $[ X N ]$ is a square matrix of full rank. For example, $N$ could be obtained by QR decomposition.

I've confirmed numerically that the above equation for $\hat\beta$ is equal to an alternative form that performs a correction on $y$ then does ordinary least squares:

$ \hat\beta = (X^TX)^{-1} X^T \left( I - \Sigma N \left( N^T \Sigma N \right)^{-1} N^T \right) y $

I'd like to prove mathematically that these two expressions for $\hat\beta$ are equal.

Any help simplifying the second expression for $\hat\beta$ would be greatly appreciated, or just anything that it reminds you of.

Paul Harrison
  • 391
  • Your terminology is a bit off- $N$ is a matrix whose columns form an orthonormal basis for the null space of $X$. The null space itself is a set of infinitely many vectors. – Brian Borchers Mar 07 '15 at 23:54
  • Thanks, this sort of terminology is something I want to get straight. I've edited the question. – Paul Harrison Mar 08 '15 at 23:10
  • In addition, I'd add "hats" to your betas in your estimated equations as your $\beta$'s are estimated here. – StatsStudent Mar 08 '15 at 23:28

1 Answers1

0

Since the covariance is positive definite we can find $S$ such that

$\Sigma = SS^T$

We have that $X^T N = 0$, so $X^T S^{-T}S^T N = 0$, so $(S^{-1}X)^T (S^TN) = 0$.

Use that if B is a basis for a null space of A then their hat matrices are related

$ I-B(B^TB)^{-1}B^T = A(A^TA)^{-1}A^T $

Choosing $A=S^{-1}X$ and $B=S^TN$, it becomes a fairly simple exercise.

Paul Harrison
  • 391