6

Suppose you have some dataset $X \in \mathbb{R}^{n \times d}, y \in \mathbb{R}^{n}$ from some distribution $\mathcal{D}$ (if convenient to obtain a closed form formula, $\mathcal{D}$ can be a Gaussian distribution or something else that is tractable).

You posit a linear model, and you compute the least squares estimate $\hat{\beta} = (X^T X)^{-1} X^T y.$

Can you say anything about the MSE of $\hat{\beta}$? Of course, I imagine that for a fixed $X$ this is difficult, but maybe you could compute the expectation of the the MSE of $\hat{\beta}$, where the randomness in this expectation is over the dataset $(X,y)$?

This feels like a standard enough problem so that I wanted to first ask if anyone knows about this or has any helpful references. Thanks!

Alan Chung
  • 211

2 Answers2

6

You haven't said what you want to take as the true value $\beta^*$ when defining the MSE; I'll take it as the value that minimises expected squared error for predicting $y$, which is the the thing that $\hat\beta$ estimates.

It's actually easier for a fixed $X$. In that setting, $\hat\beta$ is a deterministic linear function of $y$ and is unbiased for $\beta^*$. By linearity its variance (and thus also its MSE) is $$\mathrm{var}[\hat\beta] = (X^TX)^{-1}X^T \mathrm{var}[y] X(X^TX)^{-1}$$ where $\mathrm{var}[y]$ is the covariance matrix of $y$ (diagonal if the observations are independent -- you didn't specify). This can be estimated by the 'sandwich' estimator $$\widehat{\mathrm{var}}[y]= (X^TX)^{-1}X^T (X^T(y-X\hat\beta)(y-X\beta)^TX) X(X^TX)^{-1}$$

If $X$ is not fixed, you get to use the conditional variance formula $$\mathrm{var}[\hat\beta]=\mathrm{var}[E[\hat\beta|X]]+E[\mathrm{var}[\hat\beta|X]]$$ where neither term is zero. This is harder because the best 'true' $\beta$ depends on the configuration of $X$ (which is why the first term isn't zero).

Larry Wasserman's lecture notes here have some relevant bounds over distributions ${\cal D}$ with bounded support, including a bound for prediction MSE.

llllvvuu
  • 223
Thomas Lumley
  • 38,062
  • I think I might have been asking a slightly different question. I want to know the following: if $\hat{\beta}$ is computed, is there a way to analyze the population mean squared error (i.e., averaged over the distribution $\mathcal{D}(x,y)$) of $\hat{\beta}$? I think this is slightly different from the variance of $\hat{\beta}$. – Alan Chung Jul 16 '23 at 15:49
  • It's not different if $X$ is fixed; it is different if $X$ is random, though then you also have to think about what to do when $\hat\beta$ isn't defined, which has positive probability for any $n$ and any ${\cal D}$ – Thomas Lumley Jul 16 '23 at 22:37
4

It is actually easier for a fixed $X$ than for ranging over $(X, y)$. The former case is more commonly analyzed, since it doesn't require you know the distribution of $X$.

The MSE of an estimator is equal to its variance plus the square of its bias.

Under certain conditions, OLS is an unbiased estimator. The variance of OLS can be computed as such:

\begin{align*} \operatorname{Var} \, (\hat{\beta} \, | \, X) &= \operatorname{Var} \, ((X^TX)^{-1} X^{T}Y \, | \, X) \\ &= (X^TX)^{-1} X^T\operatorname{Var}(Y \, | \, X) X (X^TX)^{-1} \\ &= (X^TX)^{-1} X^T\operatorname{Var}(\epsilon \, | \, X) X (X^TX)^{-1} \end{align*}

If the errors are normally distributed with variance $\sigma^2$ then this reduces to:

\begin{align*} \sigma^2 (X^TX)^{-1} \end{align*}

Edit: I answered at the same time as @Thomas Lumley. That answer completes the part where $\operatorname{Var}(\epsilon \, | \, X)$ is unknown so we must estimate it.

llllvvuu
  • 223
  • I think I might have been asking a slightly different question. I want to know the following: if $\hat{\beta}$ is computed, is there a way to analyze the population mean squared error (i.e., averaged over the distribution $\mathcal{D}(x,y)$) of $\hat{\beta}$? I think this is slightly different from the variance of $\hat{\beta}$. – Alan Chung Jul 16 '23 at 15:49
  • The "MSE of $\hat{\beta}$" is defined as $E[(\hat{\beta} - \beta)^2]$. It sounds like you might be looking for some other metric like the "generalization error" in which case you need to update the question. – llllvvuu Jul 17 '23 at 18:41