intuition behind exchangeability property, and its use in statistical inference

Question

I'm reading "Bayesian Data Analysis" by Gelman et al., and I encountered this exchangeability property: $\{X_n\}_{n \in N}$ is exchangeable if $F_{X_1,\ldots,X_n}(x_1,\ldots,x_n)$ is symmetric in its arguments $\forall n \in N$. I understand the definition, but not the intuition behind it. Up to now I've always only encountered i.i.d. sequences of random variables. I understand the intuition behind the i.i.d. property (for example, it's a reasonable model for coin tosses, dice throws, etc.) and its usefulness in forming various kinds of confidence intervals (mean, proportions, quantiles, regression coefficients, etc.).

I'm much more at a loss with exchangeability. Obviously i.i.d. sequences are exchangeable. But which other kind of phenomena are intuitively exchangeable, and how is this property used to perform inference? I read that an exchangeable sequence is one where the probability of a specific event (for example, with $p(X_1=1, X_2=0,\ldots,X_n=1)$ where the $X_i$ are Bernoulli) doesn't depend on the order of the results. But then sampling without replacement from a urn with $n$ black marbles and $m$ white marbles (which I read can be modeled by an exchangeable sequence of Bernoulli RVs) doesn't seem intuitively exchangeable to me, because I would think that the probabilities would depend on the results of the extractions. Probably it's the conditional probabilities which depend on the extraction history, and not the joint density, but I'm still confused...I would need some intuitive interpretation of exchangeability, and one or two simple examples where we use an exchangeable, but not i.i.d, sequence of random variables to perform statistical inference.

did you have a look at http://stats.stackexchange.com/questions/3520/can-someone-explain-the-concept-of-exchangeability ? — Christoph Hanck, Dec 31 '15 at 15:49
yes, but with absolutely no offence meant to the answerer, I don't like that answer for two reasons.

If an exchangeable sequence is identically distributed, as the answer says, then https://en.wikipedia.org/wiki/Exchangeable_random_variables#Examples is wrong because here definitely the marginal distribution of each $X_i$ is not the same. To understand that answer, I would need a proof that exchangeable $\implies$ identically distributed. — DeltaIV, Dec 31 '15 at 15:56
I specify that in my first comment, I'm referring to the second example in https://en.wikipedia.org/wiki/Exchangeable_random_variables#Examples (the one about the urn with red and blue marbles). I'm writing it here, because I cannot edit my first comment anymore. — DeltaIV, Dec 31 '15 at 16:09

score 16 · Accepted Answer · answered Jan 01 '16 at 00:57

Exchangeability, loosely writing, means you can permute the indices of the random variables in the expression $F(x_1, \dots, x_n)$ without having the result of the probability calculation change. This means, basically, that you can put the observed value of, for example, $x_1$, in where $x_3$ is in the list of values and vice versa (or more complex permutations) without altering the calculated probability.

Consider an urn example; 3 black balls and 2 white balls, sampling without replacement. Now let's draw two balls; we get one white and one black. Does the probability of the sequence $(w,b)$ equal that of the sequence $(b,w)$? If so, and if this holds for all sequences and all samples, then the sequence is exchangeable, although the draws themselves are clearly not independent.

$P(b,w) = 3/5 * 1/2 = 3/10$

$P(w,b) = 2/5 * 3/4 = 3/10$.

If we see $x_1 = w$ and $x_2 = b$, and permute the indices in the probability calculation to (2,1) instead of (1,2), which means we calculate $P(b,w)$ instead of $P(w,b)$, we'll get the same numeric result. The fact that this is universally true in urn models of this sort means that the sequence of draws (from urn models of this sort) is exchangeable.

As for why we care, I can hardly do better than to point you to this paper by Bernardo (for the Bayesian perspective.) The tl;dr is that exchangeability is all that's necessary to show the existence of a probability distribution and a prior distribution on the parameter(s) of the probability distribution. So it's pretty fundamental stuff, not something you (directly) use to, e.g., help construct a particular statistical test.

To quote: "if a sequence of observations is judged to be exchangeable, then, any finite subset of them is a random sample of some model $p(x_i | \theta)$, and there exists a prior distribution $p(\theta)$ which has to describe the initially available information about the parameter [$\theta$] which labels the model."

thanks a lot, also for the paper which I'm going to print! I have still a doubt: since $F_{X_1,X_2}(x,\infty)=F_{X_1}(x)=F_{X_1,X_2}(\infty,x)=F_{X_2}(x)$, it does look like exchangeable implies identically distributed. It seems really weird to me that, in sampling without replacement, the probability distribution of the second extraction is precisely the same of the probability distribution of the first extraction. Am I missing something? PS Happy new Year!! — DeltaIV, Jan 01 '16 at 11:13
It is true that the marginal distributions are identical. More than that, the conditional distributions are also identical under permutations of the indices - $p(x_1 = b | x_2 = b, x_3 = w) = p(x_2 = b | x_1 = b, x_3 = w)$ and so forth, to continue the above example. It's weaker than i.i.d., though, as it allows $p(x_1 = b | x_2 = b) \neq p(x_1 = b | x_2 = w)$, whereas independence requires they be equal. And Happy New Year to you too! — jbowman, Jan 03 '16 at 00:17
@DeltaIV: I use this for demonstrations in class. I take an urn with, say, 7 red and 3 blue (otherwise identical) chips. I draw 1, keep it in the clsed hand, and ask what is the probability it is red. ... Then I repeat, draw one ball, without looking, and throws it out of the window. Then I draw one more, and ask what is the probability that it is red ... — kjetil b halvorsen, Feb 20 '18 at 14:13

score 1 · Answer 2 · edited Apr 13 '17 at 12:44

1

Equicorrelated random variables are another example for exchangeability but non-iid-ness. To take the simplest useful example, consider $n=3$. The correlation matrix then is $$ \begin{pmatrix} 1&\rho&\rho\\ \rho&1&\rho\\ \rho&\rho&1 \end{pmatrix}$$ As each r.v. has correlation $\rho$ (within the legitimate limits, see here) with any other r.v., it does not matter which r.v. is in the first, second or third position.

edited Apr 13 '17 at 12:44

Community

1

answered Jan 08 '16 at 08:55

Christoph Hanck

33,180

1

Hi, I don't think that equicorrelation implies exchangeability. Exchangeability means that the joint CDF function is symmetric in the $n$ arguments, and it's easy to derive equicorrelation from it. However, I don't see why the converse should be true. The information contained in the joint CDF is much more that the information contained in the correlation matrix. – DeltaIV Jan 08 '16 at 19:20
You are right - I may restore it by referring to distributions that are fully characterized by their first two moments, like the multivariate normal. – Christoph Hanck Jan 08 '16 at 19:59
de Finetti's theorem need infinite exchangeability, which in this case implies that the given $3\times 3$ matrix can be the upper left corner of a covariance matrix of the same form,< but of any size. That in turn implies that $\rho \ge 0$. – kjetil b halvorsen Feb 19 '18 at 08:48

intuition behind exchangeability property, and its use in statistical inference

2 Answers2

Linked