2

I am struggling with writing down a relationship between two variables and I am happy for each and every comment that points out, how I can overcome this problem:

Let $x\sim N(\mu,\Sigma), x \in\mathbb{R}^{N-1}$ be a vector that is multivariate normally distributed with mean parameter $\mu$ and covariance matrix $\Sigma$. $x_N$ is of such that $\sum_{i=1}^{N}x_i=1$. (Therefore: conditional on $x$ the scalar $x_N$ is deterministic defined as $1-\sum x$.) It is a standard result that $X_n$ (unconditional of $x$) as the sum of normal distributed variables is also normally distributed with mean parameter $1-\iota'x$ (where $\iota$ is a vector of ones) and variance $\iota'\Sigma\iota$. However, it is an obstacle for me to write down the joint distribution of $z:=(x,x_N)$ (which is apparently not normal due to the deterministic behavior of $x_N$ given $x$). However, I can easily sample from the joint distribution and it is not surprising to see for me that the expected value of $z=E(z)=(\mu,1-\iota'\mu)$. Sampling also gives me that the covariance between $x_N$ and $x_i$ ($i\in\{1,\ldots,N-1\}$) is what I would expect: $$cov(x_N,x_i)=cov(1-\sum_{j=1}^{N-1}x_j,x_i)=-\sum_{j=1}^{N-1}cov(x_j,x_i)$$ which is nothing but minus the sum of the $i$-th row of $\Sigma$. Can anyone help me to explain how to mathematically show that this is correct although I don't have the joint distribution of $z$?

Stefan Voigt
  • 1,330
  • I believe you mean $N$ instead of $N-1$ in the first line. Perhaps many of your difficulties have been addressed at http://stats.stackexchange.com/questions/178389? – whuber Oct 25 '15 at 19:27
  • Thank you @whuber! The $N-1$ is correct in my question, but in fact your linked question is related to mine: The support of the joint distribution $(x,x_N)$ falls into an $N-1$ dimensional subspace. However, I can not point out from the answer of @air how to come up with the covariance matrix. Can you give me a hint? – Stefan Voigt Oct 25 '15 at 19:47
  • 1
    You don't need the joint distribution to determine the covariance of $X_N$ with any other $X_i$: use the law of the unconscious statistician to determine the expectation of $g(X_N,X_i)$ from the distribution of just the $X_i, i = 1,2,\ldots,N-1$. – Dilip Sarwate Oct 25 '15 at 20:30

1 Answers1

2

Suppose that random variables $X_1,X_2,\ldots,X_{N-1}$ have a a multivariate normal distribution (jointly normal) with mean vector $\mathbf m = (\mu_1,\mu_2, \ldots, \mu_{N-1})$ and covariance matrix $C$. Define $$X_N = 1 - \sum_{i=1}^{N-1}X_i.$$ Then, $X_N$ is also normally distributed with mean $\displaystyle E[X_N] = \mu_N = 1 - \sum_{i=1}^{N-1} \mu_i$ and variance $$\sigma_N^2 = \operatorname{cov}(X_N,X_N) = \operatorname{cov}\left(\sum_{i=1}^{N-1}X_i, \sum_{j=1}^{N-1}X_j\right) = \sum_{i=1}^{N-1}\sum_{j=1}^{N-1} C_{i,j}.$$ Furthermore, the transformation $(X_1,X_2,\ldots, X_{N-1})\to (X_1,X_2,\ldots, X_{N-1}, X_N)$ is a linear transformation (well, actually an affine transformation if you want to nitpick) of jointly normal random variables, and so $(X_1,X_2,\ldots, X_{N-1}, X_N)$ also enjoys a jointly normal distribution with mean vector $\mathbf m^\prime = (\mathbf m, \mu_N) = (\mu_1,\mu_2, \ldots, \mu_{N-1}, \mu_N)$ and covariance matrix $\Sigma$ which is a bordered $N\times N$ matrix in which

  • $C$ is the $(N-1)\times(N-1)$ submatrix in the upper left hand corner,

  • $\Sigma_{N,N} = \sigma_N^2$ appears in the lower right hand corner

  • the $(N,j)$-th entry in the bottom row of $\Sigma$, which is $\operatorname{cov}(X_N,X_j)$, has value $$\Sigma_{N,j} = \operatorname{cov}(X_N,X_j) = \operatorname{cov}\left(1-\sum_{i=1}^{N-1}X_i,X_j\right) = -\left(\sum_{i=1}^{N-1}\operatorname{cov}(X_i,X_j)\right) = -\sum_{i=1}^{N-1} C_{i,j},$$ that is, $\Sigma_{N,j}$ is the negative of the sum of the entries in the $j$-th column of $C$. Put another way, the column sums of $\Sigma$ equal $0$, and of you think about it a little, so do the row sums of $\Sigma$ equal $0$.

Note that all that we have used is the fact that covariance is a bilinear function.

Dilip Sarwate
  • 46,658
  • Thank you very much! But I struggle with the point that $X_1,\ldots,X_N$ is also jointly normal distributed...because just sampling from an $N$-Dimensional normal with the given Variance matrix does not automatically fullfill that the draw is part of the $N-1$ dimensional subspace, or does it? – Stefan Voigt Oct 25 '15 at 20:33
  • Ah sorry, now I understand it...the bordered matrix is not of full rank, so strictly speaking I would not be able to draw from this distribution..correct? – Stefan Voigt Oct 25 '15 at 20:34
  • Yes, the bordered matrix has the same rank as $C$. $\sum_i\sum_j\Sigma_{i,j} = \operatorname{var}\left(\sum_{k=1}^N X_k\right) = 0$. – Dilip Sarwate Oct 25 '15 at 20:38