2

We say that in "standard" OLS regression the residuals $û$ are uncorrelated with $k$-th explanatory variable $x_k$. I know the argument can be intuitively derived from geometry of OLS. There are a lot of interesting articles in this forum on that topic with the bottom line, that the projection is orthogonal.

I am only interested, if that is also true in the following three cases:

  • we do a regression without a constant
  • $x_k$ is not exogenous
  • $E(x_{ik}' u_i) = 0$

I am not quite sure about the first one. I think I've heard, that we need the constant in the model for it to be true. The exogeneity should not matter, as far as I know, i.e. the projection is still orthogonal. And I think the third one does not matter for the question.

Marlon Brando
  • 435
  • 2
    There are sometimes substantive grounds to force a regression through the origin. That can make sense as a reason for insisting on $y = bx$. After all, the laws of physics met at introductory level often have this form. I would need a stronger story for insisting on $y = b_1 x_1 + b_2 x_2 + \cdots + b_p x_p$ and omitting a $b_0$. – Nick Cox Nov 25 '23 at 12:18
  • Hi Nick, Thanks! Not sure if I really get you right. So if I regress y on x without a constant, the projection will still be orthogonal? And what about the other two facts? I assume that exogeneity does only matter for causal interpretation, but not in terms of the construction. And the third one I am still not sure. I think applying the law of iterated expectation yields always 0, if x is exogenous. So that would bring me back to the second statement. Long story short: Is û always uncorrelated with the explanatory variables? – Marlon Brando Dec 02 '23 at 14:13
  • https://stats.stackexchange.com/questions/474102/why-is-the-correlation-between-independent-variables-regressor-and-residuals-zer

    The first answer also tells that the fact is only true, when we have a constant in the model.... not sure what to think right now....

     – Marlon Brando Dec 02 '23 at 15:29
  • 4
    Since you understand the geometry, you don't need to ask about orthogonality: you know the residuals are orthogonal to all vectors in the space spanned by the regressor variables. (That's called the Normal Equations.) What might be confusing is that when that space contains no nonzero constant vectors, orthogonality does not imply uncorrelated. – whuber Dec 05 '23 at 14:21
  • So in other words, orthogonality holds always? But $xi' ûi = 0$ only holds if we have a constant in the model? Is that not a contradiction? – Marlon Brando Dec 09 '23 at 16:13
  • I just realized that the original projection matrix P = X (X'X)^-1 X' does not involve any assumption about the error term. Is this what you mean? But that automatically implies the fact $x_i û_i = 0$ right? – Marlon Brando Dec 09 '23 at 16:24

1 Answers1

1

I am not quite sure about the first one. I think I've heard, that we need the constant in the model for it to be true.

See the following example where the residuals clearly correlate with the variable $x$ when we fit a model without constant.

set.seed(1)
x = 0:20
y = 5 + x + rnorm(21)
plot(x, y, ylim = c(0,30))
lines(x, predict(lm(y~0+x)))

example or residuals correlating with x

And I think the third one does not matter for the question.

Consider the covariance (related to the correlation)

$$cov(x,u) = E[xu] - E[x]E[u]$$

The fact that $E[xu] = 0$ does not guarantee zero correlation. In particular if $E[u] \neq 0$ which can happen when there is no intercept.

Orthogonal, as in $x \cdot u = 0$, does not mean uncorrelated, $(x-\bar{x}) \cdot (u - \bar{u}) = 0$

Sextus Empiricus
  • 77,915
  • @marlon, indeed, orthogonality always holds: For reg <- lm(y~0+x) we still have, by the OLS properties whuber mentions, that crossprod(resid(reg),x) $\approx0$. But, since the mean of the residuals is not zero here (it must be zero when we have a constant, as $\text{cst}'\hat u=0$ is the orthogonality condition for the constant, so that the sum is zero, and hence also the mean). Hence cor(resid(reg),x) is not zero. – Christoph Hanck Jan 12 '24 at 13:43