How to choose between different methods of linear regression?

Question

I find following commonly mentioned linear regression methods:

OLS is usually the default. I believe robust model is to be used to correctly handle outliers, but I am not clear about others.

What are the criteria to choose one over the other?

Edit: It is mentioned in the comments that it is a very broad question (I did not know that!). However, I would like to have a one or two lines on each of above to know the "indications" or when to use them.

OLS: default

RLM: if outliers are important and cannot be ignored.

That leaves only GLS and WLS. What would be most important reasons to use them?

This is very general, and whole books are needed for an answer. Could you limit the scope somewhat? Linear regression methods for ... ? — kjetil b halvorsen, Jul 12 '20 at 17:35
I have tried to specify that I need only a general indication on when to use each of these. — rnso, Jul 12 '20 at 17:40
OLS is useful as pedagogically to generally introduce regression and it's concepts and assumptions, but I am not sure if or when I actually have used OLS in research. Regression methods permitting creative violation of OLS regression's assumptions permit us to model the complexities with which the world behaves with more fidelity. — Alexis, Jul 12 '20 at 17:49

score 3 · Accepted Answer · answered Jul 14 '20 at 02:16

3

Your links goes to statsmodels program web pages, a software I do not know. I will assume their use of terms is the standard. A very general indication, just as a starter, what you really need is a book on regression.

OLS is the starting point, many other models can be seen as extensions or generalization. Assumptions is continuous response, linear effects and constant variance + independence (of residuals.)
GLS weakens assumption, do not assume constant variance nor independence. So you will need somehow to model the variance and covariances.
WLS is GLS but with covariances zero, so really an assumption of independence of residuals.
RLM is really a huge class of models and methods. Especially think about this for routine or automatized analyses.

answered Jul 14 '20 at 02:16

kjetil b halvorsen

77,844

I had asked earlier https://stats.stackexchange.com/questions/146077/why-not-robust-regression-everytime . But I cannot understand how to adjust options in RLM since you say that RLM is "a huge class and methods". – rnso Jul 14 '20 at 03:11
It is suggested at https://stats.stackexchange.com/questions/473603/log-transform-in-linear-regression that WLS is useful if there is a problem of Heteroscedasticity. – rnso Jul 15 '20 at 00:55
Well, that is what I tried to say, do not assume constant variance. – kjetil b halvorsen Jul 15 '20 at 01:21
as far as i know, the only difference between OLS and WLS is the weights that get multiplied as part of whitening, while no whitening is applicable for OLS. given this, i would like to know, how can assumptions become less stringent in WLS from OLS and how WLS can be useful when there is heteroscedasticity? – Naveen Reddy Marthala Mar 28 '22 at 01:19

How to choose between different methods of linear regression?

1 Answers1