8

Generalized Linear Models (GLMs) are more general than Linear Regression by construction. Nearly the same question was asked here: When to use GLM instead of LM?. However I'm not very satisfied of the different answers proposed.

I was wondering: is there a systemic way to know if GLM would be more appropriate than Linear Regression, something as straightforward as a test?

Victor
  • 1,045

3 Answers3

15

As with many other cases in statistics, the goal of finding a single test to replace one's judgement is a bad one.

There are several sources of information you can and should use while deciding: the theoretical expectation of the distribution, prior empirical work on the topic, the properties of the data (e.g. is it truncated or zero-inflated?), and the residual distributions and other diagnostics after fitting models. But there is no single, general test (or even a set of tests) that will tell you what to do.

And there cannot be one. I recognise the intuitive appeal of having a decision tree to follow when making such a choice, especially in an area that is complex and new to you. But there are few hard boundaries in the areas you need to consider, and so this decision does not lend itself well to such a workflow. You need to use judgement, and developing that will take time and practice.

mkt
  • 18,245
  • 11
  • 73
  • 172
  • 1
    I see, could you maybe give share more details on the process you use to decide? For instance by walking us through an example with real data. I'm interested to see which precise points you focus on on each step of your reasoning. – Victor Sep 14 '19 at 17:02
  • 2
    @Victor I agree it would be useful, but I'm afraid doing justice to that would require a much longer answer than is feasible for CV (it's more like a book chapter). That's because as you can see from Isabella Ghement's very helpful answer, there is a wide range of models to consider, many with different sets of assumptions and diagnostics. Frans Rodenburg's answer gives you a nice, succinct way to think about where you might want to apply GLMs. But if you want a broad overview, a book or book chapter on the topic might be best. – mkt Sep 15 '19 at 05:44
  • 1
    @Victor Also, there are great answers to more specific questions about GLMs here on CV that you might find useful: https://stats.stackexchange.com/questions/tagged/generalized-linear-model?tab=Votes – mkt Sep 15 '19 at 05:45
13

Another great answer from @mkt on this forum. Here are a few more pointers you might find useful.

GLMs include some widely used types of regression models:

  1. Binary Logistic Regression Models;
  2. Binomial Logistic Regression Models;
  3. Multinomial Logistic Regression Models;
  4. Ordinal Logistic Regression Models;
  5. Poisson Regression Models;
  6. Beta Regression Models;
  7. Gamma Regression Models.

As pointed out by @COOLSerdash in his comment, beta regression models share some features - such as linear predictor, link function, dispersion parameter - with GLMs (GLMs; McCullagh and Nelder 1989), but are NOT special cases of the GLM framework. However, I included them in the above list because of their similarity with GLMs and their practical value.

A good place to start would be to familiarize yourself with each of these types of models and when it might be used.

Binary Logistic Regression Models

These types of models are used to model the relationship between a binary dependent variable Y and a set of independent variables X1, ..., Xp.

For example, Y could represent the survival status of patients at a local hospital assessed 30 days following a surgical intervention for treating a particular disease such that Y = 1 for a patient who survived and Y = 0 for a patient who died. Furthermore, if p = 2, then X1 could represent Age (expressed in years) and X2 could represent gender. For all the subsequent examples below, it will be assumed that p = 2 and that X1 and X2 will have the same meaning as in the current example.

Binomial Logistic Regression Models

These types of models are used to model the relationship between a binomial dependent variable Y and a set of independent variables X1, ..., Xp.

For example, Y could represent the number of correct questions (out of 10) answered by patients on a questionnaire eliciting their knowledge of the symptoms associated with their disease.

Multinomial Logistic Regression Models

These types of models are used to model the relationship between a nominal dependent variable Y with more than 2 categories and a set of independent variables X1, ..., Xp.

Ordinal Logistic Regression Models

These types of models are used to model the relationship between an ordinal dependent variable Y and a set of independent variables X1, ..., Xp.

For example, Y could represent the degree of pain experienced by patients immediately after surgery, expressed on an ordinal scale from 1 to 5, where 1 stands for no pain and 5 stands for severe pain.

Poisson Regression Models

These types of models are used to model the relationship between a count dependent variable Y and a set of independent variables X1, ..., Xp.

For example, Y could represent the number of hospital days (out of 30) when patients had to use pain relieving medication following their surgery.

Beta Regression Models

These types of models are used to model the relationship between a dependent variable Y expressed as a continuous proportion taking values in the open interval (0,1) and a set of independent variables X1, ..., Xp.

For example, if the disease in question is a brain disease, Y could represent the fraction of the brain area still affected by disease 30 days post-surgery relative to the total brain area for patients who survived the surgery.

Gamma Regression Models

These types of models are used to model the relationship between a positive-valued, continuous dependent variable Y and a set of independent variables X1, ..., Xp.

For example, Y could represent the healthcare utilization costs of patients who survived up to the 30-day mark.

Hong Ooi
  • 8,249
Isabella Ghement
  • 20,314
  • 2
  • 34
  • 58
  • 1
    @Isabelle Ghement, I was wondering if the dependent variable Y is indeed positive and continuous (for instance a house price), will it always make more sense to use a Gamma Regression Model instead of a Linear Regression? – Victor Sep 14 '19 at 16:58
  • 1
    @Victor: Yes, gamma regression is often used to model the relationship between a dependent cost variable having a strongly positively skewed distribution and a whole bunch of independent variables. I added an example inspired by your comment in my answer. – Isabella Ghement Sep 14 '19 at 18:54
  • 4
    +1 This might be nitpicking but I seem to recall that beta regression isn't strictly a GLM. The authors of the betareg package write: "... the model shares some properties [...] with generalized linear models [...], but it is not a special case of this framework [...]." I'm afraid that I'm not prepared to answer exactly as to why. – COOLSerdash Sep 14 '19 at 19:04
  • 2
    Great point, @COOLSerdash - I updated my answer to address it. – Isabella Ghement Sep 14 '19 at 19:38
  • For Poisson regression see https://stats.stackexchange.com/questions/142338/goodness-of-fit-and-which-model-to-choose-linear-regression-or-poisson/142353#142353 – kjetil b halvorsen Jan 14 '20 at 23:41
7

This was a reply to @Victor's comment on @mkt's answer, but it grew rather large, and I suppose it answers the question.

The point of using a GLM is to allow different error distributions than Gaussian.

Is the data generating process continuous, with a central tendency and can it take on both positive and negative values? Then a regular LM is a decent starting point. Is the answer to any of these questions no? Then determine which error distribution could be appropriate and start with a GLM or GAM using that error distribution. Isabella's answer provides some concrete examples of when to use which distribution.

After this, you should always perform visual diagnostics. Your assumptions may be reasonable from a theoretical standpoint, but severely violated in practice.

There is no singular method, or test for this process, because even in cases where the assumptions for a normal (or really any) distribution are violated, the model could still approximate the process well.

Remember that all models are wrong. The point is to find one that is useful, and a good starting point for that is theoretical substantiation. Reserve tests only for comparison of a handful of candidate models.

(Of course, this is assuming you are using your model for inference. For prediction problems, you shouldn't be looking at goodness-of-fit at all.)

Frans Rodenburg
  • 10,925