Hows does coefficient $b_1$ change when estimating $b_1 x_1+b_2 x_2+b_3 x_3$ instead of $b_1 x_1+b_2 x_2$

Question

My question is related to [1], [2] and [3].

Assume we estimate a multiple regression,

$$ y = a + b_1x_1 + b_2x_2 + u $$

and are mainly interested in the value of $\hat{b}_1$ (lets denote this specific estimate $\hat{b}_{1; \text{model 1}}$).

If we run a different model by including an additional independent variable $x_3$

$$ y = a + b_1x_1 + b_2x_2 + b_3x_3 + u $$

we will observe a different estimate $\hat{b}_1$ (denoted as $\hat{b}_{1; \text{model 2}}$), because the answer in [1] states that

A parameter estimate in a regression model will change if a variable is added to the model that is:

1. correlated with that parameter's corresponding variable (which was already in the model), and
2. correlated with the response variable

Question:

Does there exist a closed formula for the change in the estimated coefficient $\hat{b}_1$ when including additional independent variables?

---

Edit: Assume we just include one additional indep. variable $x_3$ where all observations are known. Of course, one could run both regressions in that case, but does there exist a way to directly calculate the change in the estimated $\hat{b}_1$?

Answer 1

By the Frisch–Waugh(–Lovell) theorem and the well-know formula $\left(X^{\top}X \right)^{-1}X^{\top}y$ for the OLS estimator we have $$ \begin{align} \hat{b}_{1; \text{model 1}}&=\left( \left(M_{2}x_1 \right)^{\top}M_{2}x_1 \right)^{-1} \left(M_{2}x_1 \right)^{\top}M_{2}y =\left( x_1^{\top}M_{2}x_1 \right)^{-1} x_1^{\top}M_{2}y,\\ \hat{b}_{1; \text{model 2}}&=\left( \left(M_{2,3}x_1 \right)^{\top}M_{2,3}x_1 \right)^{-1} \left(M_{2,3}x_1 \right)^{\top}M_{2,3}y =\left( x_1^{\top}M_{2,3}x_1 \right)^{-1} x_1^{\top}M_{2,3}y, \end{align} $$ with symmetric and idempotent matrices $$ \begin{align} M_{2}&=I-\left( \mathbf{1}\,x_2 \right) \left( \left( \mathbf{1}\,x_2 \right)^{\top} \left( \mathbf{1}\,x_2 \right) \right)^{-1} \left( \mathbf{1}\,x_2 \right)^{\top},\\ M_{2,3}&=I-\left( \mathbf{1}\,x_2\; x_3 \right) \left( \left( \mathbf{1}\,x_2\; x_3 \right)^{\top}\left( \mathbf{1}\,x_2\; x_3 \right) \right)^{-1} \left( \mathbf{1}\,x_2\; x_3 \right)^{\top}. \end{align} $$ Hence, the change in the OLS estimate of $b_1$ is given by
$$ \hat{b}_{1; \text{model 2}}-\hat{b}_{1; \text{model 1}}=\left( \left( x_1^{\top}M_{2,3}x_1 \right)^{-1} x_1^{\top}M_{2,3}-\left( x_1^{\top}M_{2}x_1 \right)^{-1} x_1^{\top}M_{2} \right)y $$ or simply by $$ \left( x_1^{\top}M_{2,3}x_1 \right)^{-1} x_1^{\top}M_{2,3}y - \hat{b}_{1; \text{model 1}} $$ if you already know the value of $\hat{b}_{1; \text{model 1}}$.

Answer 2

No, there is no closed form for the the change in the estimated coefficient $\hat{b}_1$ when including additional independent variables. Why? When adding other independent variables, say $x_4, x_5, ...$, there will be some effects to other existing variables, say $x_2$. We cannot predict how it will change and how much it will affect. Here, one may face multicollinearity problem, which can be detected and solved using several methods such as variance inflation factor, principal component regression, and so on.

Answer 3

We can find both $\hat{b}_{1, \text{model } 1}$ and $\hat{b}_{1, \text{model } 2}$ are closed form (take the appropriate element of $(X^T X)^{-1}X y$) thus the difference between them is also available in closed form.

I do not think we can, in general, offer a simpler or more elegant solution than computing their differences because matrix inversion is a complex operation. Nonetheless, it is a closed form solution.