Why are "Linear" Models so Important?

Question

I am an MBA student that is taking courses in statistics. Yesterday, I attended a statistics seminar in which some graduate students presented their research on some psychology experiments (e.g. response of mice to some stimulus).

The students presented some theory behind the models they used - in particular, they showed us about "General Linear Models" (GLM). I think I was able to follow the general ideas that were discussed. It seems to me that GLM's model some function of the mean instead of directly modelling the mean - this gives GLM's the advantage of being more flexible and versatile in modelling more complicated datasets.

• The one thing that I couldn't understand however, is that why is the "Linear" part of GLM's so important? Why can't you just use/call it "General Models"?

• When I tried to raise this point in the question and answer period, someone told me that the "Linear" aspect of GLM's allow for "ease in estimation" (i.e. easier "number crunching") compared to "Non-Linear" models and that the "Linear" aspect also allows for easier interpretation of the model compared to "Non-Linear" Models.

I am also not sure what exactly is a "Non-Linear" Model. I would have thought that a Logistic Regression is a "Non-Linear" Model because the output looks non-linear and it can model non-linear relationships, but it was explained to me that any model that can be written "linearly" like "x1b1 + x2b2 + ..." is called a "Linear Model". Therefore, is something like a Deep Neural Network a "Non-Linear Model"?

I couldn't really understand why this is true - can someone please help me with this?

Answer 1

• GLMs are linear in parameters, that's why “linear”. See also Distinction between linear and nonlinear model and Why is polynomial regression considered a special case of multiple linear regression? for more explanations.
• Why would GLMs be “general models” any more than any other models? You also got the name wrong, it stands for the generalized linear models, because it is a generalization of linear regression. General linear model in fact stands for something else.
• They are important because they are simple, yet versatile. They are easy to interpret, yet very flexible, and often work remarkably well for many problems.

Answer 2

The reason that linear models are so common in statistics is that they are simple and they allow relatively simple derivation of results. These models are more general and flexible than you might imagine, since "linearity" in this context means linearity with respect to the model parameters, not linearity with respect to the explanatory variables.

In regression analysis we are seeking to infer the conditional distribution of a response variable $Y$ conditional on an explanatory vector $\mathbf{x}$. This generally means that we make an inference about the conditional expected value of the response (called the "true regression function") and we also make an inference about the conditional distribution/variability around this conditional expectation. The linearity property in a GLM is a property pertaining to the true regression function; it means that you can write the main model equation for this function in this form:

$$g(\mathbb{E}(Y|\mathbf{x}, \boldsymbol{\beta})) = \beta_0 + \sum_{i=1}^k \beta_i f_i(\mathbf{x}),$$

where $g$ is the link function in the model and $f_1,...,f_k$ are a set of functions that transform the explanatory vector $\mathbf{x}$. Observe here that linearity is with respect to the parameter vector $\boldsymbol{\beta}$, not the explanatory vector $\mathbf{x}$ (though in some cases the equation is also an affine function of the latter). Below I give some examples of linear and nonlinear regression equations, to give you a flavour of the breadth of the linear model and the types of cases that fall outside its scope.

In order to understand the "simplicity" of the linear model, you will need to learn the derivation of inference results in GLMs and other linear models. If you take some time to learn how to derive the inference results from the underlying model form you will see that there are a number of steps in the derivation that use the linear form of the model, and which become much more complicated when dealing with a nonlinear model. It is possible to derive inference results and other properties for nonlinear models, but the derivation is substantially more complicated and it often requires numerical methods where nonlinear functions are locally approximated by linear functions. If you would like to learn this in depth, I recommend you devote some time to learning the derivation of the inference results in linear regression and GLMs; there is really no substitute here for actually getting into the first-principles derivations of the inference results in the linear model to see how the linear form makes things easier.

---

Examples of linear and nonlinear model equations: The following equations all give linear models:

$$\begin{align} \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\beta}) &= \beta_0 + \sum_{i=1}^k \beta_i x_i, \\[6pt] \mathbb{E}(Y|x, \boldsymbol{\beta}) &= \beta_0 + \sum_{i=1}^k \beta_i x^i, \\[6pt] \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\alpha}) &= \alpha_0 \prod_{i=1}^k \alpha_i^{x_i}, \\[6pt] \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\alpha}) &= \alpha_0 \prod_{i=1}^k x_i^{\alpha_i}, \\[14pt] \mathbb{E}(Y|x, \alpha, \kappa) &= \alpha \sin ( 2 \pi (x + \kappa) ). \\[14pt] \end{align}$$

(As an exercise, see if you can successfully write each of these model equations in explicitly linear form.) The following equations all give nonlinear models:

$$\begin{align} \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\alpha}) &= \alpha_0 + \prod_{i=1}^k \alpha_i^{x_i}, \\[6pt] \mathbb{E}(Y|\mathbf{x}, \boldsymbol{\beta}) &= \beta_0 + \sum_{i=1}^k e^{\beta_i x_i}, \\[14pt] \mathbb{E}(Y|x, \boldsymbol{\alpha}) &= \alpha_0 x e^{-\alpha_1 x}, \\[20pt] \mathbb{E}(Y|x, \alpha, \kappa) &= e^{-\alpha x} \sin ( 2 \pi (x + \kappa) ), \\[20pt] \mathbb{E}(Y|x, \alpha, \phi, \kappa) &= \alpha \sin ( 2 \pi \phi (x + \kappa) ). \\[14pt] \end{align}$$

(As an exercise, attempt to put these model equations into explicit linear form, and see why you can't do it.) It is also worth noting that the nonlinear equations can be locally approximated by linear equations using either Taylor approximation or Fourier approximation, so while the latter are not linear, they can be approximated by linear equations reasonably well in some cases.