4

I'm just learning about Covariance and encountered something I don't quite understand.

Assume we have two random variables X and Y, where the respective joint-probability function assigns equal weights to each event.

According to wikipedia the Cov(X, Y) can then be caluculated as follows:

$${\displaystyle \operatorname {cov} (X,Y)={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-E(X))(y_{i}-E(Y)).}$$

What confuses me is the fact, that they sum only over $x_i$ and $y_i$; $i=1,...,n$ , but not $x_i$ and $y_j; \;i=1,...,n$ $j=1,...,m$ , thus many possible combinations are not calculated. In other words if we look at each calculated combination individually, we only get a $n*1$ matrix, instead of a $n*m$ matrix.

Can anyone explain this (I suppose it's rather obvious but I just don't see the reason at the moment).

meow
  • 242
  • 5
    Maybe this helps. Lets say you are interested in the relationship between height and weight in people. Covariance could give an indication of this relationship. In this case I am interested if someone with a larger height weighs more, or less. This means that I am interested in the relationship between the height and weight of each individual person, over all persons. I am not interested in the relationship between person A s weight and person B height. Hence, in the calculation both features should stem from the same observational unit, a person. The $i$ designates a particular person. – spdrnl Mar 11 '17 at 17:05
  • I was typing something along these lines, but I erased it now, because the prior comment summarizes it well. – Antoni Parellada Mar 11 '17 at 17:06

1 Answers1

5

The idea is that the possible outcomes in your sample are $i=1, \ldots, n$, and each outcome $i$ has equal probability $\frac{1}{n}$ (under the probability measure that assigns equal probability to all outcomes that you appear to be using). You have $n$ outcomes, not $n^2$.

To somewhat indulge your idea, you could compute:

$$ \operatorname{Cov}(X,Y) = \sum_i \sum_j (x_i - \mu_x) (y_j - \mu_y) P(X = x_i, Y = y_j )$$

Where:

  • $P(X = x_i, Y = y_j) = \frac{1}{n}$ if $i=j$ since that outcome occurs $1/n$ times.
  • $P(X = x_i, Y = y_j) = 0 $ if $i \neq j$ since that outcome doesn't or didn't occur.

But then you'd just have:

$$ \sum_i \sum_j (x_i - \mu_x) (y_j - \mu_y) P(X = x_i, Y = y_j ) = \sum_i (x_i - \mu_x) (y_i - \mu_y) P(X = x_i, Y = y_i ) $$

Which is what the original formula is when $P(X = x_i, Y = y_i ) = \frac{1}{n}$.

You intuitively seem to want something like $P(X = x_i, Y = y_j ) = \frac{1}{n^2}$ but that is seriously wrong.

Simple dice example (to build intuition):

Let $X$ be the result of a roll of a single 6 sided die. Let $Y = X^2$.

Recall that a probability space has three components: a sample space $\Omega$, a set of events $\mathcal{F}$, and a probability measure $P$ that assigns probabilities to events. (I'm going to hand wave away the event stuff to keep it simpler.)

$X$and $Y$ are functions from $\Omega$ to $\mathcal{R}$. We can write out the possible values for $X$ and $Y$ as a function of $\omega \in \Omega$

$$ \begin{array}{rrr} & X(\omega) & Y(\omega) \\ \omega_1 & 1 & 1\\ \omega_2 & 2 & 4 \\ \omega_3 & 3 & 9 \\ \omega_4 & 4 & 16 \\ \omega_5 & 5 & 25 \\ \omega_6 & 6 & 36 \end{array} $$

We don't have 36 possible outcomes here. We have 6.

Since each outcome of a die is equally likely, we have $P( \{ \omega_1) \}) = P( \{ \omega_2) \}) = P( \{ \omega_3) \}) = P( \{ \omega_4) \}) = P( \{ \omega_5)\}) = P( \{ \omega_6) \}) = \frac{1}{6}$. (If your die wasn't fair, these numbers could be different.)

What's the mean of $X$?

\begin{align*} \operatorname{E}[X] = \sum_{\omega \in \Omega} X(\omega) P( \{ \omega \} ) &= 1 \frac{1}{6} + 2\frac{1}{6} + 3 \frac{1}{6} + 4 \frac{1}{6} + 5 \frac{1}{6} + 6 \frac{1}{6}\\ &= \frac{7}{2} \end{align*}

What's the mean of $Y$?

\begin{align*} \operatorname{E}[Y] = \sum_{\omega \in \Omega} X(\omega) P( \{ \omega \} ) &= 1 \frac{1}{6} + 4\frac{1}{6} + 9 \frac{1}{6} + 16 \frac{1}{6} + 25 \frac{1}{6} + 36 \frac{1}{6}\\ &= \frac{91}{6} \end{align*}

What's the covariance of $X$ and $Y$?

\begin{align*} \sum_{\omega \in \Omega} \left(X(\omega) - \frac{7}{2}\right)\left( Y(\omega) - \frac{91}{6}\right) P( \{ \omega \} ) &= \left( 1 - \frac{7}{2} \right)\left( 1 - \frac{91}{6} \right) P(\{\omega_1\}) + \left( 2 - \frac{7}{2} \right)\left( 4 - \frac{91}{6} \right) P(\{\omega_2\}) + \left( 3 - \frac{7}{2} \right)\left( 9 - \frac{91}{6} \right) P(\{\omega_3\}) + \left( 4 - \frac{7}{2} \right)\left( 16 - \frac{91}{6} \right) P(\{\omega_4\}) + \left( 5 - \frac{7}{2} \right)\left( 25 - \frac{91}{6} \right) P(\{\omega_5\}) + \left( 6 - \frac{7}{2} \right)\left( 36 - \frac{91}{6} \right) P(\{\omega_6\}) \\ &\approx 20.4167 \end{align*}

Don't worry about the arithmetic. The point is that to calculate $\operatorname{Cov}\left(X , Y\right) = \operatorname{E}\left[(X -\operatorname{E}[X])(Y - \operatorname{E}[Y]) \right] = \sum_{\omega \in \Omega} \left(X(\omega) - \operatorname{E}[X]\right)\left( Y(\omega) - \operatorname{E}[Y]\right) P( \{ \omega \} ) $ you sum over the 6 possible outcomes $\omega_1, \ldots, \omega_6$.

Back to your situation...

The possible outcomes in your sample are $i=1\, \ldots, n$. Those are the outcomes you should sum over.

Matthew Gunn
  • 22,329
  • Thanks a lot, I think that really helped. I messed up the concept of probability space. As I was somehow thinking of one probability space per random variable instead of realizing that the joint probability has its own probability space. Now the notation makes sense. – meow Mar 11 '17 at 22:51
  • @meow So one common situation that occurs is if you have two independent random variables $X$ and $Y$. Then you can construct the sample space for the joint distribution as the Cartesian product of the two sample spaces. The joint probability $P(X=x,Y=y)$ would equal to the product of the individual probabilities $P(X=x)P(Y=y)$. Then you would get a $\sum_i \sum_j$ type situation where $P(X,Y) = P(X)P(Y)$. (Of course, in this situation, the covariance would automatically be zero because $X$ and $Y$ are independent.) – Matthew Gunn Mar 12 '17 at 00:18