16

When to use a generalized linear model over linear model?

I know that generalized linear model allows for example the errors to have some other distribution than normal, but why is one concerned with the distributions of the errors? Like why are different error distributions useful?

mavavilj
  • 4,109
  • 1
    If the assumption you make on the error is true, you get estimates which are (in general) the most efficient ones. Some times the distribution you impose has some nice properties... Like fitting a dummy outcome with logit or probit rather than OLS – Repmat Dec 13 '16 at 17:35
  • @Repmat Is it because one is able to relax on the linear model assumption "errors must be normally distributed" and since they don't have to, then one may be able to get a better fit? Are there any general categories of problems that are more suited for GLMs than LMs? – mavavilj Dec 13 '16 at 17:45

3 Answers3

7

A GLM is a more general version of a linear model: the linear model is a special case of a Gaussian GLM with the identity link. So the question is then: why do we use other link functions or other mean-variance relationships? We fit GLMs because they answer a specific question that we are interested in.

There is, for instance, nothing inherently wrong with fitting a binary response in a linear regression model if you are interested in the association between these variables. Indeed if a higher proportion of negative outcomes tends to be observed in the lower 50th percentile of an exposure and a higher proportion of positive outcomes is observed in the upper 50th percentile, this will yield a positively sloped line which correctly describes a positive association between these two variables.

Alternately, you might be interested in modeling the aforementioned association using an S shaped curve. The slope and the intercept of such a curve account for a tendency of extreme risk to tend toward 0/1 probability. Also the slope of a logit curve is interpreted as a log-odds ratio. That motivates use of a logit link function. Similarly, fitted probabilities very close to 1 or 0 may tend to be less variable under replications of study design, and so could be accounted for by a binomial mean-variance relationship saying that $se(\hat{Y}) = \hat{Y}(1-\hat{Y})$ which motivates logistic regression. Along those lines, a more modern approach to this problem would suggest fitting a relative risk model which utilizes a log link, such that the slope of the exponential trend line is interpreted as a log-relative risk, a more practical value than a log-odds-ratio.

AdamO
  • 62,637
2

Well, there are plenty of reasons to choose a different error distribution. But I believe that you aren't aware on why we have distributions for variables in the first place. If this is obvious to, I believe my answer is useless to you, sorry.

Why distributions are important

See, having distributions allows us to consider a model in a probabilistic, meaning we can quantify uncertainties about our model. When in stat 101 we learn that the sampling distribution of the sample mean $\bar{X} \dot{\sim} \mathcal{N}(\mu,\sigma)$ (asymptotically), we can, in a probabilistic framework, tell a lot of stuff about that estimate, like testing hypothesis, constructing confidence intervals.

Probabilistic Distributions in linear and generalized linear models

When in a linear model framework, we can basically do the same, if we know the the distribution of the error term. Why? This a result of linear combination of random variables (see this answer). But the point is, when this probabilistic structure is present in the model, we can again do sorts of stuff. Most notably, besides hypothesis testing and constructing C.I, we can build predictions with quantified uncertainty, model selection, goodness of fit testes and a bunch of other stuff.

Now why do we need GLMs specifically? Firstly, the probabilistic framework of a linear model can't handle different types of that, such as counts or binary data. Those types of data are intrinsically different them a regular continuous data, meaning its possible to have a height of 1.83 meters, but its senseless to have 4.5 electrical lights not working.

Therefore the motivation for GLMs starts with handling different types of data, primarily by the use of link functions or/and by cleverly manipulating the intended model to a linear known "framework". These needs and ideas are connected directly to how the errors are modeled by the "framework" being used.

Community
  • 1
Guilherme Marthe
  • 1,388
  • "errors" do not have distributions except in some formulations of OLS models. If you are instead talking about a distribution of $Y$ conditional on $X$, then it can be shown that some GLMs are a maximum likelihood technique using natural parameterization. However, not all GLMs are MLE, but nevertheless very useful – AdamO Dec 14 '16 at 20:11
1

There are two things we should care about,

  1. consistency,
  2. efficiency.

If we don't have 1, screw 2. But if we have 1, we would like to get 2 if possible.

If you run OLS, then it is consistent under very general assumptions about the error distribution (you just need exogeneity). However, GLS can be more efficient. This is particularly nice if you have a small sample.

Superpronker
  • 812
  • Is general linearity really only about efficiency in computation? – mavavilj Dec 13 '16 at 18:24
  • I'm talking about statistical efficiency: i.e. How many observations are required to achieve a certain level of precision (in probability). – Superpronker Dec 13 '16 at 20:33
  • @Superpronker I think a rather glaring omission is interpretability or usefulness. Along those lines, Cox has said, "It is to be stressed that the provision of exactly, or very nearly, unbiased estimators is rarely, if ever, important in its own right." In fact, I bet there are examples where a misspecified OLS model would be more efficient than a GLM. – AdamO Dec 13 '16 at 23:19
  • @AdamO, good point. Also, as OP hints at, computational efficiency: if we can avoid numerical optimization altogether then that is preferable as well. – Superpronker Dec 14 '16 at 04:54
  • @Superpronker The GLM actually has nothing to do with optimization. It just so happens that for regular exponential families, a mean-variance relationship makes it possible to do maximum likelihood with GLMs, but in general it's just an estimating equation approach. We find a root to the equation $D^TV^{-1} \left(Y - g^{-1} (\beta X) \right)$, for any old $D$ or $V$. – AdamO Dec 15 '16 at 17:18
  • I see. So you run OLS on the transformed data? – Superpronker Dec 16 '16 at 04:37