4

I'm having trouble understanding identifiability. Specifically, I'm not sure, in the following example, why $P\left(C\right)$ cannot be identified. Here's the example:

You have 2 unfair coins with the following probabilities:

$$P\left(H|C_1\right)=1/4$$ $$P\left(H|C_2\right)=3/4$$

Each coin has an equal probability of being chosen, so $P\left(C=C_1\right)=P\left(C=C_2\right)=1/2$.

My question is, given that I only observe $H$, why can't I identify $P\left(C\right)$ using basic algebra. That is, \begin{eqnarray} P\left(H\right)&=&P(C_1)P\left(H|C_1\right) + P\left(C_2\right)P\left(H|C_2\right)\\ &=& P\left(C\right)\left(P\left(H|C_1\right)+P\left(H|C_2\right)\right)\\ P\left(C\right) &=& \frac{P\left(H\right)}{P\left(H|C_1\right)+P\left(H|C_2\right)} \end{eqnarray}

In this case $P\left(C\right)$ would seem to be identifiable, so I'm not sure where I'm going wrong. Note that I'm not looking to identify $C$ itself (the actual coin), I want to know the distribution on $C$.

Phillip Cloud
  • 959
  • What does P(C) mean? Is it to represent the probability distribution for the chosen coin? In other words P(C=H)=1-P(C=T). If so how does P(C=H) relate to P(C=C1)? I think there is an algebra mistake at the point when you factor P(C) but I can't be sure because you didn't explain what it represents. – Michael R. Chernick Jul 12 '12 at 21:54
  • $P\left(C\right)$ represents the probability of picking either $C_1$ or $C_2$. – Phillip Cloud Jul 12 '12 at 22:23
  • How does one "observe" a probability?? Also, the probability of picking either $C_1$ or $C_2$ equals $1$ by the axioms. – whuber Jul 12 '12 at 22:27
  • @whuber $P\left(C\right)$ is the probability distribution over the set of possible coins. Is that correct? I'm honestly asking, I don't want you to think I'm being a smart-ass – Phillip Cloud Jul 12 '12 at 23:29

2 Answers2

8

Identifiability problems arise in parameter estimation when you have two or more different sets of parameter values that give rise to the same probabilities of observing the random variable. In that case, you can't distinguish between the different possible values of the parameters by looking at the data, no matter how much data you have. (Writing loosely here...)

In your case, I suspect you're slightly misunderstanding what you observe and what you aren't. Typically you would observe the outcome of the coin flip $H$ or $T$, and you wouldn't know the probabilities, because if you did identifiability is a moot issue (since you've no parameters to estimate.)

In this case, let us assume that you DON'T know $P(H|C_1)$, $P(H|C_2)$, and $P(C_1)$, and all you observe is some collection of heads and tails. I assume you are interested in estimating the three probabilities.

Now, note that if $P(H|C_1)=0.75$, $P(H|C_2)=0.25$, and $P(C_1)=0.25$, then:

$P(H) = P(H|C_1)P(C_1) + P(H|C_2)P(C_2) = 0.75*0.25 + 0.25*0.75 = 6/16$

But if $P(H|C_1)=0.25$, $P(H|C_2)=0.75$, and $P(C_1)=0.75$, then, skipping the detail, $P(H) = 6/16$ again. In fact, there are an infinite number of parameter values (the probabilities) that will generate $P(H)=6/16$, or any other value you care to pick. $P(C_1)$ can be any probability, and you can find a $P(H|C_1)$ and $P(H|C_2)$ that will generate any specific $P(H)$. Consequently, no sequence of heads and tails will allow you to estimate $P(C_1)$. $P(C_1)$ is not identifiable.

jbowman
  • 38,614
0

To add to @jbowman great answer, I believe this identifiability problem can be regarded also as a $\textit{Label Switching}$ problem.

The likelihood can be written as

$$p_{1}f_{C=1}(H)+p_{2}f_{C=2}(H) = \sum_{k=1}^{2}p_{k}f_{C=k}(H)$$

where $p_{1}+p_{2}= P(C=C_{1})+P(C=C_{2})=1$ and $f_{C=i}(H)=P(H|C=C_{i})$ with $i=1,2$.

And the problem of identifiability arises, because the likelihood is not invariant under permutations, i.e. we can change the labels of the components and acquire a likelihood that is equivalent to the original

$$p_{2}f_{C=2}(H) + p_{1}f_{C=1}(H)=\sum_{k=1}^{2}p_{k}f_{C=k}(H)$$

Fiodor1234
  • 2,224