Conditional probability for autocorrelated binary random variables

Question

In this answer, @whuber outlines conditional probabilities for correlated binary random variables. In some recent exploration, I have been playing with inference for sequences of binary random variables and have discovered that for some values of the coin's bias ($q$) and autocorrelation ($\rho$), these conditional probabilities fail to be valid.

Here is a simple example. The probability of observing a 1 in the next flip given the last flip was also a 1 is

$$ P(1 \vert 1) = q + \rho(1-q) $$

I've gone through the exercise of plotting this probability for various $q$ and $\rho$

For some values of $\rho$ and $q$ the conditional probability is negative, and hence I suspect is for some reason undefined. The simplest example I could think of is as follows: Suppose the coin's bias was $q=0$ and the correlation was $\rho=-1$. Obviously, the correlation would imply the next flip would have to be 1, but since the bias is 0 this is not possible. This is an extreme example where the conditional probability is not defined precicely because $q=0$ in the denominator of Bayes rule, but as we can see the conditional probability is negative even when $q>0$.

I was hoping someone could explain to me why these conditional probabilities are turning out to be negative, and in general what bounds I would need to enforce on the bias and correlation in order for this not to happen.

Answer 1

The short answer (using the notation in the reference) is that $\alpha$ can only take on certain values given $q$ which then restricts the values for $\rho$ and $\text{Pr}(1|1)$ so everything is fine.

Suppose the length of a sequence is 100 with 80 1's and 20 0's. Here $q=80/100$. For a lag of 1 $\alpha$ is the proportion of 1's followed immediately by a 1. The smallest value that $\alpha$ can be is 59/99 (as there are 99 pairs to consider). This results in a minimum value of $\rho$ being

$$\rho=\frac{\alpha-q^2}{q(1-q)}=\frac{\frac{59}{99}-\left(\frac{80}{100}\right)^2}{\frac{80}{100}\cdot\frac{20}{100}}=\frac{-109}{396}\approx-0.275253$$ So

$$Pr(1|1)=q+\rho(1-q)=\frac{80}{100}+\left(\frac{-109}{396}\right)\left(1-\frac{80}{100}\right)=\frac{295}{396}\approx0.744949$$

The minimum value of $\alpha$ for lag 1 will occur for any sequence with 80 1's and 20 0's that has no two successive 0's.

Addition

Here's maybe a handwaving argument when there are more 1's than 0's for lag 1:

If there are $n_0$ zeros and $n_1$ ones with $n_1\geq n_0$, then $q=n_1/(n_0+n_1)$ and the minimum and maximum values of $\alpha$ are $(n_1-n_0)/(n_1+n_0-1)$ and $(n_1-1)/(n_0+n_1-1)$, respectively. The minimum occurs when there are no two consecutive zeros and the maximum occurs when all the zeros and ones are completely separate in the sequence.

This means that the minimum and maximum values that $\rho$ can take are $1-\frac{n}{n_1(n-1)}$ and $\frac{-n_0^3-n_0^2 n_1+n_1^2}{n_0 n_1 (n_0+n_1-1)}$, respectively.

This in turn means that $\text{Pr}(1|1)$ and take on values between (and including) $\frac{((n_1-n_0) (n_0 + n_1))}{(n_1 (n_0 + n_1-1))}$ and $\frac{(n_1-1) (n_0+n_1)}{n_1 (n_0+n_1-1)}$.

For a specific example, suppose $n_0=107$, $n_1=395$ with sequence length being $n_0+n_1=500$. We have $q=0.786$ with $\alpha$ taking on values from $0.573146$ to $0.785571$. $\rho$ can take on values from $-0.26545$ to $0.785571$. And $\text{Pr}(1|1)$ can range between $0.729194$ to $0.999454$.