Covariance of $\frac{x}{\|x\|}$ for Gaussian x?

Question

Suppose $x$ is sampled from zero-centered Gaussian with $d\times d$ covariance matrix $\Sigma$.

• is there a name for distribution of $y=\frac{x}{\|x\|}$?

• is there a closed form expression for covariance of $y$?

Answer 1

I think that the distribution of $y=\frac{X}{||X||}$ where $X\sim \mathcal{N}(0, \Psi)$ is the projected normal distribution.

I came across a similar problem and used an approximation that works well in most cases. My approximations also work for the case of non-zero mean.

For reference, here is a previous question of mine, that I responded, for the case of $X\sim \mathcal{N}(\mu, \mathbf{I}\sigma^2)$ (i.e. isotropic noise). See the approach used in that question of approximating each element $i,j$ of the second-moment matrix (i.e. $\mathbb{E}\left(\frac{X_i X_j}{||X||^2}\right)$) as a ratio of quadratic forms that uses a symmetric $A$ in the numerator that is specific to each $i,j$.

For your question, we can use the exact same approach of approximating the elements of the second-moment matrix with that same ratio of quadratic forms, but generalizing the answer above to the case where $X\sim \mathcal{N}(\mu, \Psi)$ where $\Psi$ is any symmetric positive semi definite matrix. For this generalized quadratic form, a second order Taylor approximation can be found in this article, section 3.1.

Using the formulas in the paper above for each matrix $A$ corresponding to each element $i,j$ of the second moment matrix (as done in my other question I linked), and converting the resulting equations to matrix formulas, we get the following approximation:

\begin{equation} \label{nonIso} \mathbb{E}\left( \frac{XX^T}{||X||^2} \right) \approx \frac{\mu_N}{\mu_D} \odot \left( 1 - \frac{\Sigma^{N,D}}{\mu_N\mu_D} + \frac{Var(D)}{\mu_D^2} \right) \end{equation}

where the terms are defined as follows: \begin{equation} \begin{split} & \mu_N = \Psi + \mu \mu^T \\ & \mu_{D} = tr(\Psi) + ||\mu||^2 \\ & Var(D) = 2 tr(\Psi^2) + 4 \mu^T \Psi \mu \\ & \Sigma^{N,D} = 2 \left[\Psi \Psi + \mu \mu^T \Psi + \Psi\mu \mu^T \right] \end{split} \end{equation}

Note that while $\mu_N \in \mathbb{R}^{d\times d}$ and $\Sigma^{N,D} \in \mathbb{R}^{d\times d}$, $\mu_D \in \mathbb{R}$ and $Var(D) \in \mathbb{R}$. Also, importantly, $\odot$ is element-wise multiplication, and the ratios between matrices that appear there are also element-wise.

Maybe for your case of 0 mean you would be able to apply the same approach and find an exact formula for the ratios, instead of this approximation.

Finally, that is the second-moment matrix, not really the Covariance matrix. You can subtract the outer product of the expected value of the projected Gaussian, to obtain the covariance the following way: $$ Cov\left( \frac{X}{||X||} \right) = \mathbb{E}\left( \frac{XX^T}{||X||^2} \right) - \mathbb{E}\left( \frac{X}{||X||} \right) \mathbb{E}\left( \frac{X}{||X||} \right)^T$$

I would suppose that because your variable is centered and its symmetric, the $\mathbb{E}\left( \frac{X}{||X||} \right) = 0$, though I can't say for sure.

I tried this approximation on simulated data and it works quite well.

Answer 2

By applying dherera's formula to trace-normalized zero-centered Normal with diagonal $d\times d$ covariance matrix $H$, we get the following estimate for the corresponding covariance matrix $H_p$ of projected Normal.

$$H_p = c H -2H^2\\ c=(1+2\operatorname{Tr}(H^2)) $$

When $H$ eigenvalues follow power-law decay with $p=1.1$ we can estimate eigenvalues of $H_p$ using Monte-Carlo and see the following fit:

For $1<p<2$, this estimate is slightly biased.

For instance, when $p=1.1$ using "adjusted" formula by changing $c$ to be $c=(1+1.7\operatorname{Tr}(H^2))$ gives a slightly nicer fit:

Error measures mean relative residual squared, tail error only considers smallest $d/2$ eigenvalues.

Notebook