5

Under the condition of simple linear regression model ( $Y_i = \beta_0 + \beta_1X_i + \epsilon_i$) ordinary linear estimators($\hat{\beta_0}$ and $\hat{\beta_1}$) have minimum variance among all linear estimators.

To prove OLS estimator $\hat{\beta_1} = \sum{k_iy_i}$ has minimum variance we start by setting $\tilde{\beta_1} = \sum{c_iy_i}$ and we show that variance of $\tilde{\beta_1}$ can only be larger than $\beta_1$ if $c_i \neq k_i$.

Similarly, I am trying to prove that $\hat{\beta_0}$ has minimum variance among all unbiased linear estimators, and I am told that the proof starts similarly.

I know that the OLS estimator is $\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x}$.

How do I start the proof: by constructing another linear estimator $\tilde{\beta_0}$? Is this a linear estimator $\hat{\beta_0} = c\bar{y} - \hat{\beta_1}\bar{x}$?

Quazi Irfan
  • 252

2 Answers2

2

The result you are trying to prove is called the Gauss-Markov theorem, and there are a number of available proofs you can find with a quick internet search. It can be proved using some simple matrix algebra. Since your goal is to prove the theorem yourself, I will not give you the full proof, but hopefully I can get you started, and give some general tips how to approach the problem. The usual starting point for the proof would be to assume a linear estimator of the form:

$$\hat{\boldsymbol{\beta}} = \mathbf{A} \mathbf{Y} \quad \quad \quad \quad \quad \mathbf{A} = (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{B}.$$

The ordinary least squares (OLS) estimator occurs in the case where $\mathbf{B}=\mathbf{0}$, and other linear estimators occur in the case where $\mathbf{B} \neq \mathbf{0}$. Now, from here you should be able to use some matrix algebra to obtain expressions for the mean and variance of the estimator, using the assumption that $\mathbf{Y}$ follows the form of the linear regression model. Unbiasedness requires that the mean be equal to the true parameter vector, and this should give you some requirement on the matrix $\mathbf{B}$. Using this condition, you should be able to simplify your variance expression. You then need to show that this variance expression is minimised when $\mathbf{B} = 0$, so that the OLS estimator is the MVLUE.

Ben
  • 124,856
  • Thank you. But we have not gotten into Multiple Linear Regression yet. I need to know how can I solve it without using matrix notation. – Quazi Irfan Feb 04 '20 at 18:43
2

You can start by expressing $\hat{\beta_0}$ as a linear combination of $y_i$, similar to $\hat{\beta_1} = \sum{k_iy_i}$:

$\hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x} = \frac{1}{N}\sum{y_i}-\sum{k_iy_i}\bar{x}=\sum{(\frac{1}{N}-k_i\bar{x})y_i}=\sum{l_iy_i}$

Evgenia
  • 336
  • Thanks. I was able to figure it out myself. But this is what I was looking for. Also, can you please shade some light on why the estimator I proposed is not linear estimator? – Quazi Irfan Feb 11 '20 at 03:07
  • 1
    It is, but it helps to write it out in terms of $y_i$ to see that it is very similar to $\hat{\beta_1}$. In the proof of the Gauss-Markov theorem using matrix notation there is no distinction between $\beta_0$ and $\beta_1$, they are lumped together in one $\beta$ matrix. – Evgenia Feb 15 '20 at 19:05