1

If :

  • OLS estimates are BLUE (best linear unbiased estimator)
  • OLS = MLE
  • MLE is used for GLM regression

Does this also mean that all regression coefficients from GLM are also BLUE? Or is this only correct for simple linear regression? If the first order conditions of MLE and OLS are identical, is MLE as efficient as OLS i.e. are they both BLUE Is OLS estimator the only BLUE estimator?

stats_noob
  • 1
  • 3
  • 32
  • 105
  • 1
    First of all, your reasoning logical chain is problematic. It sounds like "because I have a cat, the cat is black and female, you also have a female cat, then your cat must be black as well". Secondly, the statement "OLS estimates are BLUE" is also wrong -- it is well known that OLS is not BLUE if the error is heteroscedastic, in which case WLS is BLUE. Similarly, "OLS = MLE" is also clearly wrong -- OLS is MLE only if the error are i.i.d. Gaussians. – Zhanxiong Oct 31 '23 at 21:02

2 Answers2

3

Parameter estimates in OLS depend linearly on the observations, via the hat matrix. That's your "L" in "BLUE".

In GLM, the relationship is not linear any more. Thus, GLM parameter estimates cannot be a best linear unbiased estimator. QED.

Dave
  • 62,186
Stephan Kolassa
  • 123,354
  • What does the L in GLM describe? I thought there is linear relationship? – stats_noob Oct 31 '23 at 20:35
  • Yes, the relationship between predictors and a function of the distribution parameters is linear, but via a link function, e.g., via a log link. You can consider OLS as having an identity link function, and then the relationship between the observations and the parameter estimates is linear. – Stephan Kolassa Oct 31 '23 at 20:40
  • I disagree that the noninear link function is what disqualifies the GLM MLEs as being linear. Nothing stops me from estimating logistic regression parameters using that same $\hat\beta = (X^TX)^{-1}X^Ty$ as in OLS, which would make for a linear estimator, yet the nonlinear link function is there. Likewise, if we have an identify link yet do MLE with an assumed Gamma likelihood, the MLE would be nonlinear (I assume, though I confess that I haven't checked this). – Dave Oct 31 '23 at 20:46
1

Break down what "BLUE" means.

BEST (minimum-variance)

LINEAR

UNBIASED

ESTIMATOR

The MLE estimators for GLMs are estimators, sure, so that covers E.

In general, MLEs are not unbiased. For instance, most logistic regressions will have biased coefficient estimates. Thus, general GLM MLEs could, at best, be BLE.

The LINEAR refers to the estimator being a linear combination of the outcome, so some matrix times the $y$ vector. This would be a closed-form solution to the GLM MLE, not all of which have closed-form solutions (again, such as logistic regressions).

Finally, the BEST refers to having the smallest variance among some class of estimators, linear and unbiased estimators in the case of the Gauss-Markov theorem. With both the L and the U not holding in general for GLM MLEs, the class to which BEST would apply is to estimators that need not be linear and might be biased, so basically anything. As constants have zero variance, any constant is a zero-variance estimator of GLM coefficients, thus having lower variance than an MLE that can vary, depending on the data. (By a "constant" estimator, I mean that you predict the same values for the coefficients, regardless of the data. This is silly, sure, but it is an estimator.)

So GLM MLEs need not be unbiased, need not be linear, and need not be the best.

So you wind up with just the letter E from BLUE, that the GLM MLEs are estimators.

Fortunately, maximum likelihood estimators have a number of other desirable properties, hence their common use.

Dave
  • 62,186