2

Consider the two models $ (a) y = X\beta + u $ where $X$ is $n \times K$ and (b) $y = Z\gamma + \omega $ where $Z$ is $n \times r$. Under classical assumptions (and $Z$ and $X$ are non-stochastic) if model (a), that is $y = X\beta + u$ is the true model, show that $E(\sigma^{2}_{\omega}) \geq \sigma^2_u$ and explain the implication of your result.

I can do it in two cases when $r>k$ (overfitting) or when $r<k$ (underfitting).

Is there any short way of doing this, without cases (because for both overfitting and underfitting variance is biased)? So, what I am asking is that can they be clubbed so that result doesn't depend on relationship between $r$ and $k$?

E. Sommer
  • 1,307
  • 7
  • 20
Elina Gilbert
  • 499
  • 3
  • 13
  • If you are asking whether the variance of $w$ is expected to be greater than the variance of $u$, shouldn't you write $\mathrm E [\sigma^2_w] > \mathrm E[\sigma^2_u]$? – ahorn Aug 14 '19 at 10:53
  • No, because a) is a true model (resembling population) and b) is misspecified model, direction of misspecification is not given. That's why I took cases. But since in both the cases expected variance will be more than true variance (biased estimator). So, my ques is can these be clubbed ie. Can we show this without considering whether r is bigger or smaller to K. If so, how can we do it? – Elina Gilbert Aug 14 '19 at 11:52
  • E. Sommer understood what I was asking. They made the appropriate edit to the notation for that inequality. – ahorn Aug 14 '19 at 12:46
  • What do you mean by the "variance is biased"? If you estimate $\sigma^2$ by $s^2$ then the last random variable is unbiased for $\sigma^2$. – Bertrand Aug 14 '19 at 16:15

2 Answers2

1

If the model $ (a) y = X\beta + u $ is true, then $\omega = u + X\beta - Z\gamma $ in $(b)$, and the claim follows.

Bertrand
  • 3,351
  • 1
  • 9
  • 24
1

From a) we obtain $y - \hat{y}_x = M_xy = M_xu$ and from b) $y - \hat{y}_z = M_zy = M_zX\beta + M_zu$, where $M_x$ and $M_z$ are the residual-maker matrices (the ones from the accepted answer of the question that you linked) and $\hat{y}_x$ and $\hat{y}_z$ are the predicted values of y based on the who linear models.

The estimate of the variance is $s_x^2 = \frac{(y - \hat{y}_x)^T(y - \hat{y}_x)}{n-k}$ for the first model and $s_z^2 = \frac{(y - \hat{y}_z)^T(y - \hat{y}_z)}{n-r}$ for the second.

Thus $E(s_z^2) = \frac{E[(M_zX\beta + M_zu)^T(M_zX\beta + M_zu)]}{n-r} = \frac{1}{n-r}( E[(M_zX\beta)^T(M_zX\beta)] + E[(M_zX\beta)^T(M_zu)] + E[(M_zu)^T(M_zX\beta)] + E[(M_zu)^T(M_zu)]) = \frac{1}{n-r}( E[(M_zX\beta)^T(M_zX\beta)] + 2E[(M_zX\beta)^T(M_zu)] + E[(M_zu)^T(M_zu)]) = \frac{1}{n-r}( E[||M_zX\beta||_2^2] + 2E[\beta^TX^TM_zu] + E[(M_zu)^T(M_zu)]),$

as $(M_zX\beta)^T(M_zu) = (M_zu)^T(M_zX\beta) = \beta^TX^TM_z^TM_zu = \beta^TX^TM_zu \hspace{.2cm} \in \mathbb{R}.$

Beacuse $Z, X -$ fixed $||M_zX\beta||_2^2$ and $\beta^TX^TM_z$ are constants.

Thus $$E(s_z^2) = \frac{1}{n-r}(||M_zX\beta||_2^2 + 2\beta^TX^TM_zE(u) + E(u^TM_zu)) =$$ $$= \frac{||M_zX\beta||_2^2}{n-r} + 0 + \frac{E(u^TM_zu)}{n-r} = \frac{||M_zX\beta||_2^2}{n-r} + \sigma^2 \neq \sigma^2 \hspace{.2cm} in \hspace{.1cm}general.$$

Grada Gukovic
  • 1,278
  • 4
  • 11