3

The frisch-waugh-lovell theorem uses the following line in the proof:

$\hat\beta_2 = (X_2' M_{1}X_2)^{-1}X_2'M_{1}y$

How do we know

$(X_2' M_{1}X_2)^{-1}$

is invertible?

falconry
  • 31

2 Answers2

2

I think that we can show that the matrix is invertible if the full regressor matrix has full column rank, but please check my proof.

We are looking at a regression with $k_1+k_2$ regressors (counting a possible constant term) having a sample of size $n\geq k_1+k_2$. We asuume that the full regressor matrix $X = [X_1:X_2]$ has full-column rank equal to $k_1+k_2$. This means that $X_1$ has column rank $k_1$ and $X_2$ has column rank $k_2$.

Let $M_1 = I_{n} - P_1,\,\, P_1 = X_1(X_1'X_1)^{-1}X_1'$, the residual maker and orthogonal projection matrices respectively, symmetric and idempotent both. We want to show that the $n \times k_2$ matrix $M_1X_2$ has full column rank $k_2$, so that its Grammian matrix $[(M_1X_2)'(M_1X_2)] = X_2'M_1X_2$ is invertible.

Assume that $M_1X_2 = W$ has not full column rank. This means that at least two columns of $W=M_1X_2$ are linearly dependent, say columns $w_1 = x_{21}- P_1x_{21}$ and $w_2= x_{22} - P_1x_{22}$.
If they are linearly dependent it means that for some real scalar $a$ we have

$$w_1 = aw_2 \implies x_{21}- P_1x_{21}= ax_{22} - P_1ax_{22}$$

$$\implies x_{21}- ax_{22} = P_1(x_{21} - ax_{22})$$

So if $w_1$ and $w_2$ are linearly dependent, we have obtained that the Projection matrix $P_1$ must act as the identity matrix on the column vector $x_{21}- ax_{22}$. But in order to do so, this column vector must be included in the column space of $X_1$.

But this implies that some linear combination of the regressors in $X_1$ equals the linear combination $x_{21}- ax_{22}$ of the regressors in $X_2$... which in turn implies that the full regressor matrix $X$ has not full column rank, and so $X'X$ is not invertible to begin with.

So what I think I have proved is the following:

If the full regressor matrix $X$ has full column rank, then the matrix $M_1X_2$ as defined above has also full column rank.

Alecos Papadopoulos
  • 58,953
  • 1
    +1 I believe you, because this theorem is an immediate consequence of an easily proven generalization to arbitrary partitions of the set of explanatory variables. (I describe this generalization geometrically at http://stats.stackexchange.com/a/113207/919 and algebraically, in terms that look similar to yours, at http://stats.stackexchange.com/a/46508/919.) – whuber Mar 24 '17 at 18:22
  • I think it should be "But in order to do so, this column vector must be in the column space of $X_1$" instead of "But in order to do so, this column vector must be inlcuded in the regressors used to form $P_1$". – statmerkur Jan 10 '23 at 10:46
  • @statmerkur Thanks for the suggestion, I elaborated on it. – Alecos Papadopoulos Jan 11 '23 at 02:32
  • +1 Though, I think there is no need to state that "and so $X′X$ is not invertible to begin with." since this was not used anywhere else in your proof. – statmerkur Jan 13 '23 at 08:49
  • @statmerkur Indeed, but it is needed in order for the reader to understand that invertibility of these two matrices goes together, so if we have a "regular" linear regression least-squares estimation, FWL is also available without further worries. – Alecos Papadopoulos Jan 14 '23 at 14:33
2

With $P_1 = X_1\left(X_1^\top X_1\right)^{-1}X_1^\top$ and $M_1 = I - P_1$, $X = \begin{pmatrix} X_1 & X_2\end{pmatrix}$ can be written as $$ X = \left(M_1 + P_1 \right)\begin{pmatrix} X_1 & X_2\end{pmatrix} = \begin{pmatrix} X_1 & M_1X_2+P_1X_2\end{pmatrix}. $$ Now, assuming that $X$ has full column rank means $$ Xv = 0 \implies v = 0 $$ or, with $v = \left(v_1^\top, v_2^\top\right)^\top$, $$ X_1v_1 + \left(M_1X_2 + P_1X_2\right)v_2=0 \implies v_1=v_2 = 0 $$ or $$ X_1\left(v_1 + \left(X_1^\top X_1\right)^{-1}X_1^\top X_2v_2\right) + M_1X_2v_2 = 0 \implies v_1=v_2 = 0, $$ i.e., the columns of $\begin{pmatrix} X_1 & M_1X_2\end{pmatrix}$ are linearly independent.
A fortiori, $M_1X_2$ has full column rank, implying that $\left(M_1X_2\right)^\top M_1X_2 = X_2^\top M_1 X_2$ is positive-definite, thus invertible.

statmerkur
  • 5,950