Population parameters of a regression

Question

So this has really been bothering me and I was hoping for a (simple!) explanation if possible.

Suppose I've specified a linear regression model: $$ Y = \beta_0 + \beta_1 X + \epsilon $$ And an alternative: $$ Y = \beta_0 + \beta_1 X + \beta_2 X^2 + \epsilon $$ And I'm trying to estimate the $\beta$s, say perhaps through OLS (the exact method I don't think is relevant).

My question is: what is the exact interpretation of the $\beta$s I am trying to estimate?

The confusion arises from the fact that the population values of $\beta_1$ in either specification are presumably different, and this doesn't couch with my understanding of the population coefficients.

I had always interpreted the $\beta$s as the partial derivative of $X$ on $Y$ 'in reality'. That is, if you were to change X holding other regressors constant the change in the expected value of Y. By providing a better and better specified model, one ensured that the estimate of $\beta_1$ became more accurate (by separating out correlated variables in the error term).

This was important to my understanding; $\beta_1$ was not contingent on the specification of my model -it remained an invariant feature of the population - but rather the estimator we had for $\beta_1$ (b1) changed and became more or less accurate depending on the model.

All well and good, but this interpretation doesn't quite work in the example above. Suppose that the relationship between $X$ and $Y$ is curvilinear. If you were restricted to only include $X$ and not any higher order polynomials, then presumably the $\beta_1$ that would best describe the change in $E[Y]$ given a change in $X$ would be different than if you were to allow for higher order polynomials (in specification 2).

So say, for arguments sake, the DGP was $$ E[Y] = 1 + 10 X - 2 X^2 $$ where $0<X<2$ to ensure the polynomial doesn't influence too heavily. In this case should the true value of $X$ in specification 1 be 10? Or, to fit it to that DGP when $X^2$ is not specified should it be ~6?

It seems if it is the latter my understanding that the population coefficients do not depend on the specification go up in smoke! Please help!

Answer 1

The problem is with this:

I had always interpreted the betas as the partial derivative of X on Y 'in reality'

That's not always true in a model with interactions or various other forms of complexity.

Take a simpler example. Assume your model is $$ E[Y] = \beta_0 + \beta_1 X + \beta_2 Z + \beta_{12} XZ $$ Here the partial derivative of $E[Y]$ with respect to $X$ is $\beta_1 + \beta_3 Z$. Put another way, $\beta_1$ is only the partial derivative of $Y$ with respect to $X$ when $Z = 0$. Your model is a special case of this one.

The population marginal effect of X (the partial derivative you're talking about) is indeed one of the things you're interested in modeling with this regression. But think of it as just a happy coincidence when this quantity corresponds to particular model parameter. Generally speaking, it won't.

Answer 2

Your understanding is correct--provided we look at the model in the right way.

Because the question concerns interpreting a predictive model, we may focus on its predictions and ignore the error term. The example is sufficiently general that we might as well address it directly, so consider a model of the form

$$Y = \beta_0 + \beta_1 X + \beta_2 X^2.$$

This can be viewed as the composition of two functions, $Y = g(f(X)),$ where

$$f:\mathbb{R}\to \mathbb{R}^3,\quad f(x) = (1, x, x^2)$$

and

$$g:\mathbb{R}^3\to \mathbb{R},\quad g((x,y,z)) = \beta_0 x + \beta_1 y + \beta_2 z = (\beta_0,\beta_1,\beta_2)(x,y,z)^\prime.$$

This figure (which suppresses the unvarying first coordinate) depicts the graph of $1 + 10y - 2z$ as a blue planar surface, shows hypothetical data as red points, and plots the graph of $x\to (x, x^2)$ as a black curve. The points all lie along this curve and the planar surface, which is fit to the points, contains the curve. The following discussion distinguishes between moving about in the plane (which is described by the partial derivatives of $g$) and motion constrained to the curve (which is described by the partial derivatives of the composite function $g\circ f$.)

It is indeed the case that the betas are the partial derivatives of $g$ with respect to its arguments:

$$\beta_0 = \frac{\partial g}{\partial x},\ \beta_1 = \frac{\partial g}{\partial y},\ \beta_2 = \frac{\partial g}{\partial z},$$

all of which are constant (because $g$ is a linear transformation). In this sense, it is indeed correct to understand the betas as partial derivatives.

However, the partial derivatives of $Y$ with respect to $X$ are obtained via the Chain Rule from those of $g$ and those of $f$:

$$\frac{\partial Y}{\partial X}(X) = Dg(f(X)) Df = (\beta_0, \beta_1, \beta_2) (0,1,2X)^\prime = \beta_1 + 2\beta_2 X.$$

The function $f$ captures the fact that the three variables in the model--the constant, $X$, and $X^2$--are not functionally independent: the third is determined by the second. This lack of independence means that $X$ and $X^2$ cannot be changed separately, the way unrelated variables $X$ and $Z$ could be changed in a model of the form $Y = \beta_0 + \beta_1 X + \beta_2 Z$. In general, this is exactly what it means for any model to be "curvilinear."

In practice, $f$ is realized by the dataset itself: a separate column of values $X^2$ has to be created (either explicitly by the user or internally in response to a nonlinear model formula) out of other data columns, in this case that of $X$. The function $g$--specifically, its coefficients $(\beta_0,\beta_1,\beta_2)$--is what least squares regression estimates. By separating the nonlinear behavior ($f$) from the linear behavior ($g$) in this fashion, least squares techniques can fit nonlinear functional forms.

Only by considering these two aspects of the model--$f$ and $g$--can the coefficients be properly and fully interpreted.