4

If the random variables $X_1,...,X_n$ are I.I.d. when conditioned on an unknown parameter $\theta$, where $p(\theta)$ represents our prior beliefs about $\theta$, it follows from the commutativity of the product that $X_1,...,X_n$ are exchangeable.

In my professor's lecture notes, he goes on to write that because $X_1,...X_n$ are exchangeable, it holds that $X_1,...X_n$ are dependent. I'm having a hard time understanding why this is the case.

What if I rolled a dice three times? Then the outcomes are exchangeable, I.e. $P[(4,5,6)]=P[(5,4,6)]=P[(6,4,5)]$, yet the three rolls are independent. Am I missing something?

David
  • 1,226

3 Answers3

7

The phrase is true because if they weren't dependent, you can't learn about unsampled values. That is, learning $X_1,...,X_n$ does not improve your prediction for $X_{n+1}$ no matter how large $n$ is. Mathematically "independence" is saying

$$p(X_{n+1}=x\mid X_1,\ldots,X_n,I)=p(X_{n+1}=x\mid I)$$

In your example the dice rolls are not independent, provided the prior is not a discrete mass at a single value. To prove this in the standard case, take probability that a six comes up, using a uniform prior. The above probability is equal to the beta binomial distribution with expected value $$E(X_{n+1}\mid X_1,\ldots,X_n,I)=\frac{y+1}{n+2}$$ where $y=X_1+\ldots+X_n$ This clearly depends on $X_1,\ldots,X_n$, for all $n>1$ meaning the exchangeable data are dependent.

Another way to think is "independent" means that "posterior predictive" = "prior predictive".

User1865345
  • 8,202
probabilityislogic
  • 24,971
  • 5
    So as long as there is some uncertainty about $\theta$, the random variable $X_1$ is not independent of the random variable $X_2$ because when we know $x_1$, it tells us something about $\theta$, which tells us something about $X_2$, essentially. – David Nov 20 '19 at 13:56
  • 2
    spot on @david! – probabilityislogic Nov 21 '19 at 00:20
  • The dice rolls are independent by definition. The posterior probability on the rolls is not independent as you outline. In nature there is no uncertainty on $\theta$. It is the researcher who is uncertain about $\theta$ and by extension the posterior probability exhibits dependence. – Anirban Mukherjee Jun 05 '23 at 13:06
2

The statement:

"X1,...Xn are exchangeable $\Rightarrow$ that X1,...Xn are dependent"

is false in general case and the example you give in your question gives a correct counter-example.

Maybe, your professor wanted to outline that the converse of :

" X1,...Xn are independent $\Rightarrow$ X1,...Xn are exchangeable "

is false (which is different from the above proposition). For an example of counterexample for this last statement, see the example developed in another answer : https://stats.stackexchange.com/a/3523./14346

beuhbbb
  • 5,043
2

Repeated dice rolls are not independent by default

Three dice rolls are independent if and only if you have certainty about their probabilities before making the rolls. For example, if it's certain that you're rolling a single fair die with P(1)=P(2)=P(3)=P(4)=P(5)=P(6)=1/6, then the dice rolls are independent.

However, in the general case, if you don't have certainty about the probabilities, they are dependent events conditioned on a hidden variable i.e. the nature of that die. You'd expect the results to be correlated - if the first 99 rolls show a six 50 times, then that's probably not a fair die and the probability of rolling a 6 in the 100th roll is much larger than 1/6.

Peteris
  • 393
  • The dice rolls are independent. It's your inference that is not. Else, the i.i.d. in a process description will never make sense because the samples from the process may always be informative of the parameter set and therefore dependent. Then what does the independent in i.i.d. stand for? – Anirban Mukherjee Jun 05 '23 at 13:10