Why the OLS underestimates the variances of the coefficients

Question

CONSEQUENCES OF HETEROSCEDASTICITY

$\textbf{1}$. The presence of heteroscedasticity does not make the OLS estimates of coefficients biased, but it causes the variances of OLS estimates to increase.

$\textbf{2}$. The presence of heteroscedasticity causes the OLS to ${\color{Red} {\text{underestimate}}}$ the variances of the coefficients.

---

I don't understand why the OLS to underestimate the variances of the coefficients in $\textbf{2}$.

The following is my thoughts :

Let $\mathbf{X}$ has full column rank， $$\text{the homoscedasticity model} (1):\begin{cases} \mathbf{y}=\mathbf{X} \boldsymbol{\beta}+\varepsilon \\ E(\varepsilon)=\mathbf{0}, \operatorname{Var}(\varepsilon)=\sigma^{2} \mathbf{I} \end{cases};$$

$$\text{the heteroscedasticity model} (2):\begin{cases} \mathbf{y}=\mathbf{X} \boldsymbol{\beta}+\varepsilon \\ E(\varepsilon)=\mathbf{0}, \operatorname{Var}(\varepsilon)=\sigma^{2} \mathbf{V} \end{cases},\text{where} \mathbf{V} \text{ is diagonal but with unequal diagonal elements.}$$

$\\$ When $\text{the heteroscedasticity model} (2)$ is ture, then the weighted least-squares estimator of $\hat{\boldsymbol{\beta}}_{WLS}$ is an unbiased estimator of $\boldsymbol{\beta}$ and $$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{WLS})=\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{V}^{-1} \mathbf{X}\right)^{-1}.$$

$\\$ The ordinary least-squares estimator $ \hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$ is no longer appropriate in model $(2)$.If the ordinary least squares is used in this case, the resulting estimator $\hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$ is still unbiased. However, the ordinary least-squares estimator is no longer a minimum variance estimator. That is, the covariance matrix of the ordinary least-squares estimator is $$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{OLS})=\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{V} \mathbf{X}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1}$$ and the covariance matrix of the weighted least-squares estimator $\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{V}^{-1} \mathbf{X}\right)^{-1}$ gives smaller variances for the regression coefficients. So the OLS should overestimates the variances of the coefficients.

Answer 1

We need to note that $\sigma^{2}$ is an unknown positive constant and $\mathbf{V}=diag(\lambda_{1},\cdots,\lambda_{n})$($\lambda_{i}>0$ are not all the same; $\sigma^{2}=0$ rarely occurs in real-world data).

Assuming Model$(2)$ is true, if we misuse the formula$$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{OLS})=\sigma^{2}(\mathbf{X}^{\prime}\mathbf{X})^{-1}$$ which only holds in Model $(1)$ to calculate the variances of $$\hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$$ in Model $(2)$, then we will inevitably get the false result: $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (1)}\right)=\sigma^{2}(\mathbf{X}^{\prime}\mathbf{X})^{-1}$$since the ture value should be $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (2)}\right)=\sigma^{2}\mathbf{(X^{\prime}X)^{-1}} \mathbf{X^{\prime}VX} \mathbf{(X^{\prime}X)^{-1}}.$$

Under the assumption that all eigenvalues $\lambda_{i}$ of $\mathbf{V}$ are not less than 1,then $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (1)}\right)\leq\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (2)}\right).$$

In fact, It's very easy to explain $\mathbf{X}^{\prime} \mathbf{(V-I_{n})} \mathbf{X}\ge \mathbf{0}$ under such assumption. Note that both $\mathbf{X}^{\prime} \mathbf{V} \mathbf{X}$ and $\mathbf{X}^{\prime} \mathbf{X}$ are positive definite matrices, since $\mathbf{X}$ is full column rank.

Actually,whether the true variance of the estimator $\hat{\boldsymbol{\beta}}_{OLS}$ for $\boldsymbol{\beta}$ in Model (2) is overestimated or underestimated by the ture variance of the estimator $\hat{\boldsymbol{\beta}}_{OLS}$ for $\boldsymbol{\beta}$ in Model (1) depends on the values of $\lambda_{i}$.

Answer 2

The OLS assumes there is no heteroscedasticity, i.e. $V=I$, thus $\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{O L S}\right)$ is not the one you mentioned, instead, $\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{O L S}\right)=\sigma^2\left(\boldsymbol{X}^T \boldsymbol{X}\right)^{-1}$. However if the assumption is incorrect, i.e. the heteroscedasticity exists, $V\neq I$, this OLS formula will underestimate the variances of the coefficients.

Answer 3

Couple of notes:

• Weighted least squares uses a weight matrix $W$ and the variance of the parameters is $\sigma^2 (X^TWX^T)^{-1}$.
• No particular choice of $W$ will bias $\beta$, and so classes of these estimators can be compared in terms of their variance.
• Choosing $W=V^{-1}$ is a natural choice and, in fact, is the best linear unbiased estimator according to the Gauss-Markov theorem. As such, it's true that, if the variance matrix is anything other than $\sigma^2 I$, then $\sigma^2 (X^TV^{-1}X)^{-1} = \sigma^{*2}(X^TX) + \text{something positive}$ . Note I have to put a $^*$ on the second variance because it's a different parameter.
• The actual variance $V$ of a heteroscedastic model is rarely known. For correlated data analysis, the usual process of generalized least squares (GLS) has to do with estimating $V$ with $\hat{V}$ using the EM algorithm and some assumptions about the covariance structure. You can extend this problem to consider estimating a heteroscedastic variance matrix (with 0 off diagonal entries) using residual analyses, splines, or the like. Like the Fisher Behren's problem, there's no guarantee that this estimator is optimal, and there are cases with the OLS parameter variance is less than (or greater than) any given GLS procedure.

Answer 4

OLS will split up the observations into two orthogonal subspaces. See: Why are the residuals in $\mathbb{R}^{n-p}$?

The homoscedasticity assumption of OLS is equivalent to the assumption that the observations are spherically symmetric distributed and that the partition of sum of squares of the projections onto the two spaces is related to the dimensions of the two spaces.

This assumption is false when the errors are heteroscedastic. Depending on the orientation of the error distribution relative to the space of the model, the relationship of the two variances can be smaller or larger. So it depends and the estimates of the variances of the coëfficiënts can both be overestimated as well as underestimated.

Below is a simple example that illustrates the different cases. The situation is for a model

$$y_i \sim N(\mu x_i, \sigma_i^2)$$

and we have $x_1,x_2 = 1,2$ and we vary $\sigma_i$.

The images show each 100 simulations of samples of size two, $y_1,y_2$ and their orthogonal projection onto the line $\hat{y}_2 = 2 \hat{y}_1$, which is the OLS estimate.

• In the first image the errors are homoscedastic, and the OLS will correctly assume that the distribution of the residual has the same variance as the distribution of the estimate.
• In the second and third images the errors are heteroscedastic.
  • In the second image, when $\sigma_2 < \sigma_1$ the variance of the residuals is relatively larger than the variance of the estimate. OLS will overestimate the variance of the coëfficiënts.
  • In the third image, when $\sigma_2 > \sigma_1$ the variance of the residuals is relatively smaller than the variance of the estimate. OLS will underestimate the variance of the coëfficiënts.