Best OLS estimators

Question

Hi i am stuck on this one, the question is related to Gauss-Markov theorem:

Consider a general alternative to the OLS estimator that is also a linear unbiased estimator, say ${\tilde \beta}$. Outline a proof that the OLS estimator b is better, in a well defined sense, than ${\tilde \beta}$.

For both estimators b and ${\tilde \beta}$ i have to use the fact that $E(UU') = \sigma^2I$.

I have shown already that $ b= \left(\mathbf {X}'\mathbf {X}\right)^{-1}\mathbf {X}'\mathbf {y} = \left(\mathbf {X'}\mathbf {X}\right)^{-1}\mathbf X'(\mathbf {X}\mathbf {\beta}+u)$

$=\left(\mathbf {X'}\mathbf {X}\right)^{-1}\mathbf X'\mathbf {X}\mathbf \beta+(X'X)^{-1}X'u$

$=\beta+(X'X)^{-1}X'u$

Theorem proved for linearity, now

$E(b)= E[ \beta+(X'X)^{-1}X'u]=\beta $ as $E[u]=0$

Theorem proved for Unbiasedness.

Variance-covariance matrix of b: $Var(b)=E(b-\beta)(b-\beta)'$ using again $E(UU') = \sigma^2I$

$=E(b-\beta)(b-\beta)'=\sigma^2(X'X)^{-1}$

in order to show that b is a better estimator that ${\tilde \beta}$ i need to follow the same reasoning still using the fact that $E(UU') = \sigma^2I$ and then conclude that b has the minimum variance of all estimators thus it is the best, that is

$Var(b)\leq Var(\tilde \beta)$ i guess.

I am not sure how to properly compare the two estimators so if someone could give me a hint it would be much appreciated.

Answer 1

We can compute the variance-covariance matrice of b and $\tilde \beta$ and hence compare their variances in order to tell which one has the smallest variance. (Best estimator)

Knowing tha $E(U)=0$

$E(b-\beta)(b-\beta)'= E(\beta+(X'X)^{-1}X'u-\beta)(\beta+(X'X)^{-1}X'u-\beta)'$

$=E[(X'X)^{-1}X'u][(X'X)^{-1}X'u]$

$=E[(X'X)^{-1}X'uu'X(X'X)^{-1})$ recall that $E(uu')= \sigma^2I$

$=\sigma^2(X'X)^{-1}X'IX(X'X)^{-1}$

$=\sigma^2(X'X)^{-1}$

So it has been shown that b is unbiased as $E(b_i)=\beta_i$ and therefore $E(b)=\beta$ but also linear as $b=(X'X)^{-1}X'y$, $b=Ay$ with $A=(X'X)^{-1}X'$

Let $\tilde \beta$ a general linear unbiased estimator, $\tilde \beta\equiv(A+C)y=[(X'X)^{-1}X'+C)y$

$=[(X'X)^{-1}X'+C)(X\beta+u)$

$=\beta+(X'X)^{-1}X'u+CX\beta+Cu$, we require $CX=0$ $\forall$ $X$ for unbiasedness

$E(\tilde \beta)=E[\beta+(X'X)^{-1}Xu + CX\beta+Cu]$ with $CX=0$ required and $E(u)=0$

$=E(\beta)$

Hence $\tilde \beta$ is also unbiased and $\tilde \beta=b+Cy$ is linear

Variance-covariance matrix of $\tilde \beta$: $E(\tilde \beta-\beta)(\tilde \beta-\beta)'=E[(X'X)^{-1}X'u+Cu)[(X'X)^{-1}Xu+Cu)']$

$=E[(X'X)^{-1}X'uu'X(X'X)^{-1}+(X'X)^{-1}X'uu'C']+Cuu'X(X'X)^{-1}+Cuu'C'$

using the fact that $E(uu')=\sigma^2I$ which simplifies to

$E(\tilde \beta-\beta)(\tilde \beta-\beta)'= \sigma^2(X'X)^{-1}+\sigma^2(X'X)^{-1}X'C'+\sigma^2CX(X'X)^{-1}+\sigma^2CC'$

we know that $CX=0$ hence $(CX)'=X'C'=0$ so we have

$E(\tilde \beta-\beta)(\tilde \beta-\beta)'=\sigma^2(X'X)^{-1}+\sigma^2CC'$

$=\sigma^2[(X'X)^{-1}+CC']$ , Recall that $E(b-\beta)(b -\beta)'=\sigma^2(X'X)^{-1}$

Thus $E(\tilde \beta-\beta)(\tilde \beta-\beta)'=E(b-\beta)(b -\beta)'+ \sigma^2CC'$

We can conclude that $Var(b)\leq Var(\tilde \beta)$ which means that b is the best estimator.