0

I am trying to Derive the Estimated Variance of Regression Coefficients. I am struggling with the algebra.

Here is the model:

$$y = X\beta + \epsilon$$ $$\hat{\beta} = (X^TX)^{-1}X^Ty$$

First, we find out the expectation:

$$E(\hat{\beta}) = E((X^TX)^{-1}X^Ty) = E((X^TX)^{-1}X^T(X\beta + \epsilon)) = \beta + (X^TX)^{-1}X^TE(\epsilon)$$

Next, we find out the squared expectation:

$$E(\hat{\beta}^2) = E((X^TX)^{-1}X^Ty((X^TX)^{-1}X^Ty)^T) = E((X^TX)^{-1}X^Tyy^TX(X^TX)^{-1})$$

Substituting $y = X\beta + \epsilon$ gives:

$$E(\hat{\beta}^2) = E((X^TX)^{-1}X^T(X\beta + \epsilon)(X\beta + \epsilon)^TX(X^TX)^{-1}) = \beta\beta^T + E((X^TX)^{-1}X^T\epsilon\epsilon^TX(X^TX)^{-1})$$

Assuming that the errors are uncorrelated and homoscedastic, this simplifies to:

$$E(\hat{\beta}^2) = \beta\beta^T + \sigma^2(X^TX)^{-1}$$

Finally, we can evaluate Var($\hat{\beta}$) = E($\hat{\beta}^2$) - E($\hat{\beta}$)^2 :

$$\text{Var}(\hat{\beta}) = \beta\beta^T + \sigma^2(X^TX)^{-1} - \beta^2 = \sigma^2(X^TX)^{-1}$$

Is this the correct derivation?

  • Seems good for the most part. I would replace $\beta^2$ and $\hat{\beta}^2$ with $\beta\beta^\top$ and $\hat{\beta} \hat{\beta}^\top$ though. – angryavian Mar 13 '24 at 04:30
  • 2
    What do you mean by beta-regression? Your model seems to be ordinary least squares regression. Beta regression is about modeling a beta distributed variable. – Sextus Empiricus Mar 13 '24 at 08:36
  • "Assuming that the errors are uncorrelated and homoscedastic" -- for beta regression this is, in general, not the case. – wzbillings Mar 13 '24 at 14:10
  • sorry i meant OLS ... the beta coefficients –  Mar 13 '24 at 14:59
  • 1
    You made it too verbose -- it is a straightforward application of $\mathrm{Var}(A\xi) = A\mathrm{Var}(\xi)A^\top$ for non-random $A$ and random vector $\xi$. What you did is like rediscovering this well-known formula for this special case. – Zhanxiong Mar 13 '24 at 16:04
  • This is a FAQ: see https://stats.stackexchange.com/search?q=regression+coefficient+variance+matrix+formula – whuber Mar 13 '24 at 19:52

0 Answers0