How does the covariance matrix of the predictors in multiple regression relate to the matrix $(\mathbf{X}^T \mathbf{X})^{-1}$?

Question

Note in advance, due to my question previously being marked as a duplicate, that the question I ask here is concerned with the relationship between $(\mathbf{X}^T \mathbf{X})^{-1}$ and $\text{Cov}[\mathbf{X},\mathbf{X}]$, not the relationship between $(\mathbf{X}^T \mathbf{X})^{-1}$ and $\text{Cov}[\mathbf{X},\mathbf{y}]$ which is covered here.

The matrix formulation of multiple regression for $n$ observations is $$ \mathbf{Y} = \mathbf{X}^T \beta + \varepsilon, $$ where the error $\varepsilon$ has finite variance $\sigma^2$. Let $\mathbf{b}$ be the estimated coefficients found when we solve the multiple regression problem with least squares.

In Theorem 4.3 of the book Econometric Analysis by William H. Greene it says that asymptotically $\mathbf{b}$ is distributed as $$ \mathbf{b} = \mathcal{N}\bigg(\beta,\frac{\sigma^2}{n}Q^{-1}\bigg), $$ where $Q$ is defined in equation 4.19 as \begin{align*} Q := \text{plim}_{n \to \infty} \frac{\mathbf{X}^T \mathbf{X}}{n}. \end{align*} which is a positive definite matrix. Suppose we write the covariance matrix of $\mathbf{X}$ as $$ \text{Cov}[\mathbf{X},\mathbf{X}] = \begin{bmatrix} \sigma_{X_1 X_1} & \sigma_{X_1 X_2} & \dots \\ \sigma_{X_2 X_1} & \sigma_{X_2 X_2} & \dots \\ \vdots & & \ddots \\ \sigma_{X_n X_1} & \dots & \dots & \sigma_{X_n X_n} \end{bmatrix}, $$ where $$ \sigma_{X_i X_j} = E[(X_i-E[X_i])(X_j-E[X_j])], $$

I want to know how the matrix $(\mathbf{X}^T \mathbf{X})^{-1}$ relates to the covariance matrix $\text{Cov}[\mathbf{X},\mathbf{X}]$.

For example, if we modify the elements of $\mathbf{X}$ to make them more collinear, the off-diagonal elements of $\text{Cov}[\mathbf{X},\mathbf{X}]$ will increase in magnitude and $(\mathbf{X}^T \mathbf{X})^{-1}$ will become 'harder' to invert. But I am looking for a precise mathematical expression that relates the two matrices.

Can $(\mathbf{X}^T \mathbf{X})^{-1}$ be expressed in terms of $\text{Cov}[\mathbf{X},\mathbf{X}]$ somehow? E.g., can the elements of $(\mathbf{X}^T \mathbf{X})^{-1}$ be written in terms of the elements of $\text{Cov}[\mathbf{X},\mathbf{X}]$?

Answer 1

Maybe I don't understand your question. but if you have centered $X$, then $\Sigma=X^TX/n$.

Hence, $(X^TX)^{-1}=n^{-1}\,\Sigma^{-1}$.

> X <- scale(MASS::Boston[, 1:3], scale = FALSE)
> Cov <- function(X) ((nrow(X) - 1) / nrow(X)) * cov(X)
> Cov(X)
           crim        zn     indus
crim   73.84036 -40.13648  23.94492
zn    -40.13648 542.86184 -85.24385
indus  23.94492 -85.24385  46.97143
> (t(X) %*% X) / nrow(X)
           crim        zn     indus
crim   73.84036 -40.13648  23.94492
zn    -40.13648 542.86184 -85.24385
indus  23.94492 -85.24385  46.97143
> solve(Cov(X)) / nrow(X)
               crim            zn         indus
crim   3.207972e-05 -2.742849e-07 -1.685125e-05
zn    -2.742849e-07  5.093748e-06  9.383968e-06
indus -1.685125e-05  9.383968e-06  6.769460e-05
> solve(t(X) %*% X)
               crim            zn         indus
crim   3.207972e-05 -2.742849e-07 -1.685125e-05
zn    -2.742849e-07  5.093748e-06  9.383968e-06
indus -1.685125e-05  9.383968e-06  6.769460e-05