3

Gauss–Markov_theorem states that OLS estimator is a BLUE estimator. My doubt is can there be any other linear estimator, other than OLS, which is also a BLUE estimator?

After going through the proof of why OLS is a BLUE estimator, I feel that only OLS estimator can be the BLUE estimator. Unbiased Linear Estimators from any other techniques should essentially yield the same result as from OLS technique for them to be BLUE.

I hope I am not making any blunders in assuming so.

honeybadger
  • 1,562
  • 2
  • 12
  • 27
  • 4
    The article you link to starts with "the Gauss–Markov theorem, named after Carl Friedrich Gauss and Andrey Markov, states that in a linear regression model in which the errors have expectation zero and are uncorrelated and have equal variances, the* best linear unbiased estimator (BLUE) of the coefficients is given by the ordinary least squares (OLS) estimator, provided it exists."* – Henry Aug 06 '17 at 19:26
  • The part Henry quotes gives some immediate clues about what to vary to get something that isn't OLS... – Glen_b Aug 07 '17 at 05:28

4 Answers4

6

When the conditions for linear regression are met, the OLS estimator is the only BLUE estimator. The B in BLUE stands for best, and in this context best means the unbiased estimator with the lowest variance.

If the regression conditions aren't met - for instance, if heteroskedasticity is present - then the OLS estimator is still unbiased but it is no longer best. Instead, a variation called general least squares (GLS) will be BLUE.

Rachel Kogan
  • 291
  • Why is the OLS estimator the only BLUE estimator? If you look at the statement of the theorem, it's saying that the variance of some other estimator minus the variance of the OLS estimator is positive semi-definite. If the OLS estimator was the only BLUE estimator, then we would expect it to be positive definite. I'm not saying that you're wrong, but it would be nice to have some justification. – mlstudent Oct 29 '18 at 01:20
  • 2
    The OLS estimator does not need to be the only BLUE estimator. For example, the maximum likelihood estimator in a regression setup with normal distributed errors is BLUE too, since the closed form of the estimator is identical to the OLS (but as a method, ML-estimation is clearly different from OLS.). The Gauss–Markov Theorem however tells you that in the class of linear unbiased estimators you don't have too look further than OLS, since every other estimator in this class can not do better under the assumptions. – chRrr Jan 25 '19 at 11:01
  • do you mean generalized least squares? – rep_ho Jan 25 '19 at 11:35
  • Sorry this is not correct for a subtle reason. When variance is known, Gauss Markov theorem establishes that the weighted least squares solution is BLUE - and independent data have identity weights so the solution is OLS. However, if variance is unknown, I don't believe any such result is known. This can be seen as a generalization of the Fisher Behrens problem. GLS is an elegant solution to the problem, but we don't know if it's optimal and actually it's not at all a linear estimator - it's an EM algorithm that involves iteratively estimating the variance as well. – AdamO Feb 24 '23 at 23:34
4

The Gauss-Markov Theorem states that if a linear regression model fulfils the assumptions of the classical linear regression model the ordinary least squares estimator is the best linear unbiased estimator (BLUE).

You can find a good overview of the Gauss-Markov Theorem here:

https://economictheoryblog.com/2015/02/26/markov_theorem

Here you find the assumptions of the classical linear regression model:

https://economictheoryblog.com/2015/04/01/ols_assumptions

In order for OLS to be BLUE one needs to fulfill assumptions 1 to 4 of the assumptions of the classical linear regression model. The following website provides the mathematical proof of the Gauss-Markov Theorem. That is, it proves that in case one fulfills the Gauss-Markov assumptions, OLS is BLUE.

https://economictheoryblog.com/2016/02/05/proof-gauss-markov-theorem

Olaf Yagmur
  • 41
3

Suppose there were two different Best linear unbiased estimators, $\hat\beta_1$ and $\hat\beta_2$, with (necessarily) the same mean and variance $\sigma^2$. The average of the two would also be a linear unbiased estimator, and it would be Better. Its variance would be $$(1/2)^2\times (\sigma^2+\sigma^2+2\tau^2)$$ where $\tau^2$ is the covariance of the two. Since the estimators are different (by assumption), the covariance is less than the variance of each estimator. So the variance of the average is less than either component estimator

Thomas Lumley
  • 31
  • 1
  • Do you need to merge accounts with https://stats.stackexchange.com/users/249135/thomas-lumley ? – AdamO Feb 24 '23 at 23:19
0

I slightly disagree with the existing answers, thought it is a matter of how you view the question.

The other answers say that, if you calculate a linear and unbiased estimator that has variance equal to that of the OLS solution, you must get the OLS solution.

That’s fine, but also keep in mind that, when the errors are $iid$ Gaussian (which is an additional assumption on top of the Gauss-Markov assumptions), minimizing square loss in OLS is equivalent to maximum likelihood estimation of the coefficients.

Yes, this gives the same estimator, but that’s the point. There are multiple ways to view getting to the BLUE.

(I’m actually not convinced the other answers prove the uniqueness of the BLUE.)

Dave
  • 62,186
  • The Gauss Markov theorem makes no assumption of normality. – AdamO Feb 24 '23 at 23:13
  • @AdamO Definitely not, but if you’re willing to make some additional (common) assumptions, there is another way to view the BLUE. – Dave Feb 24 '23 at 23:23
  • It's an interesting point. With non-normal data, you could have an MLE that's not a linear estimator per se, and it could beat the BLUE estimator by bounds and leaps... but OLS would still be the best of the linear options. Something to keep in mind but not particularly exactly an answer to the question - although many people forget the L when they speak of BLUE. – AdamO Feb 24 '23 at 23:30