41

Let's say we know that A is independent of B, or mathematically:

$$P(A|B) = P(A)$$

Then how come we can't say the following is necessarily true: $$P(A|B,C) = P(A|C)$$

If the outcome of B doesn't have an effect on the outcome of $A$, then why would the outcome of $B$ AND $C$ have an effect on $A$ that is different than the effect of the outcome of just $C$?

Is there perhaps a simple counterexample to illustrate this? For example, let's say $A$ and $B$ are the event of picking an ace of spades from a deck of cards (each has their own deck of cards).

Nova
  • 699
  • 5
  • 6
  • 1
    In words: "For more than two events, pairwise independence does not imply mutual independence." – Stef Oct 11 '22 at 08:46

6 Answers6

84

My favourite example is a chessboard: if you pick a point uniformly then the row, column, and colour are pairwise independent.

Suppose A is colour, B is row, and C is column. Then P(A="white"|B)=0.5 and P(A="white"|C)=0.5 for any value of B, but P(A="white"|B,C) is either 0 or 1 for any values of B and C.

Thomas Lumley
  • 38,062
  • 3
    Lovely example! – user67724 Oct 08 '22 at 18:06
  • 13
    In particular, applying this to a 2x2 chessboard results in the example where $A,B,C$ are all uniformly random bits, but correlated so that any pair is independent and the third is their XOR. (The example in guy's answer.) – usul Oct 08 '22 at 20:58
28

Imagine $A$ and $B$ are independent Bernoulli random variables with success probability 1/2, and let $C = AB + (1 - A)(1 - B)$ (if you like, $A$ is the event that a coin lands heads, $B$ is the event that another coin lands heads, and $C$ is the event that both coins land the same side up). Then $P(A = 1 \mid C = 1) = 1/2$ but $P(A = 1 \mid B = 1, C = 1) = 1$.

A bit more generally, a common situation where this is not true is when $(A, B, C)$ form a v-structure, i.e., where $A \rightarrow C \leftarrow B$, where $A$ and $B$ are marginally independent but where $B$ can provide information about $A$ if both cause $C$.

guy
  • 8,892
18

Let's continue your example with two decks of cards, supposing that one card is chosen randomly from each deck. You have proposed that $A$ is the event that the ace of spades is chosen from the first deck and that $B$ is the event that the ace of spades is chosen from the second deck.

Now let $C$ be the event that the two cards from the two decks are of different colors. Then $$P(A|C) = P(A) = 1/52$$ but $$P(A|B,C) = 0$$ because $A$, $B$ and $C$ cannot simultaneously be true.

Note that any two of $A$, $B$ and $C$ are independent but the three events are dependent when taken together.

Gordon Smyth
  • 12,807
15

There are two situations how this can happen.

Collider bias

A common way how the situation can occur is collider bias.

An example is in the image below. In this example, intelligence and appearance are statistically unrelated (there's not even an effect in this case as with the coin example below). But conditional on the sum of intelligence and appearance the two become correlated.

example of bias

Invariant effect

With collider bias both A and B have an effect on C. In some special cases, the situation may also appear when B has a causal effect on A, but it does not appear as a statistical dependency.

The underlying reason why this counterintuitive situation can happen is because: When we speak about statistical independence between A and B then it doesn't necessarily mean that A and B have no functional/causal relationship.

Let's say that we flip two fair coins $B$ and $C$, and assign values $-1$ and $1$ to the sides of the coins. Let's define $A := B \times C$.

 B   C   A := B*C
 1   1   1
 1  -1  -1
-1   1  -1
-1  -1   1

Now $B$ does influence the outcome $A$ but $A$ is not statistically dependent on $B$ alone. Namely we have that $P(A = 1) = P(A = -1) = 0.5$ independent of the value of $B$.

Variants of this case occur when $B$ is a variable that influences $A$ while leaving the distribution of $A$ unchanged. So this case happens when $B$ has an effect on $A$ that leaves the distribution of $A$ invariant. E.g. it happens when $B$ switches labels of a fair coin or dice.

This seems like a contrived case, but it is also a bit about a particular interpretation of probability. When we flip a coin, does it matter whether we start with heads up or tails up? Statistically speaking there might not be a relationship. But in the underlying physical process the starting position of the coin has (according to theory) an influence on the position of the coin after the flip. The situation may, after all, be considered deterministic; but we describe it as probabilistic because we have no complete knowledge about the begin state of the entire system and all the parameters that are involved in the outcome.

Sextus Empiricus
  • 77,915
  • 2
    IIRC, there was another post where you mentioned about the collider bias, the first time I got acquainted with it. +1. – User1865345 Oct 08 '22 at 18:21
  • 1
    @User1865345 it might be the post from which I copied the image https://stats.stackexchange.com/a/579222/164061 – Sextus Empiricus Oct 08 '22 at 18:25
  • Thanks for the answer! In the collider example, it would help to say with $A$, $B$, and $C$ are. – Dahn Oct 09 '22 at 10:20
  • I like the idea of collider bias but I don't agree with your characterization of the second situation. I don't think it is correct to argue that "B does influence A" and "in some sense A is not independent of B" when in fact A and B are unambiguously mathematically independent. I don't accept the apparent implication that "statistical independence" is a flawed concept that misses some important type of dependence. – Gordon Smyth Oct 09 '22 at 22:14
  • @GordonSmyth the second situation relates to the idea that 'dependence' can be an ambiguous term. On the one hand you have 'statistical dependence' on the other hand you have 'causal dependence'. In the second example A := B*C is defined as a function of B and by definition dependent on B. All other things equal, B will change the value of A. I am not sure what there is to not accept. The example clearly shows a causal relationship, namely A := XOR(B,C), while A and B can be statistically independent if B and C are Bernoulli variables with p=0.5. – Sextus Empiricus Oct 09 '22 at 22:35
  • https://stats.stackexchange.com/questions/286741/is-there-an-example-of-two-causally-dependent-events-being-logically-probabilis – Sextus Empiricus Oct 09 '22 at 22:39
  • Discussing the possibility of a functional relationship between A and B that depends on an unobserved third variable is all to the good. In the relationship A = B*C, B is useful for predicting A if C is observed but useless if it is not. Great. Your answer would be improved if you could describe this situation in clear unambiguous terms without introducing new definitions for "dependence". Terms like "in some sense" or "clearly dependent" or "other things equal" or "causal dependence" confuse the issue IMO. There is only one definition of independence. – Gordon Smyth Oct 10 '22 at 01:30
  • I really like this answer. This quote goes to the heart of the matter: "The underlying reason why this counterintuitive situation can happen is because: When we speak about statistical independence between A and B then it doesn't necessarily mean that A and B have no functional/causal relationship." – Vincent Oct 11 '22 at 07:58
5

I quote Joseph Blitzstein's Introduction to Probability (2019 2 edn), p. 65.

Example 2.5.11 (Independence doesn't imply conditional independence).

My friends Alice and Bob are the only two people who ever call me on the phone. Each day, they decide independently whether to call me that day. Let A be the event that Alice calls me next Friday and B be the event that Bob calls me next Friday. Assume A and B are unconditionally independent with P(A) > 0 and P(B) > 0.

However, given that I receive exactly one call next Friday, A and B are no longer independent: the call is from Alice if and only if it is not from Bob. In other words, letting C be the event that I receive exactly one call next Friday, $P(B|C) > 0$ while $P(B|A,C) = 0$, so A and B are not conditionally independent given C.

4

As @Sextus Empiricus (the ancient skeptic!) said, collider bias phenomenon in epidemiology is a great exemplar of why marginal independence of two random variables does not necessarily imply their conditional independence, given a third random variable.

Sackett (1978)'s example is a classic for the collider bias. In his paper, $A =$ having locomotor disease, and $B = $ having respiratory disease. Let $C$ denote the event that a person is hospitalized (for whatever reason). He observed that among hospitalized patients, there was a strong association between $A$ and $B$, whereas there was no association between $A$ and $B$ in the overall population (mix of both $C$ and $\bar C$). If $C$ denotes hospitalization, then we see that $P(A|B,C) \ne P(A|C)$ and $P(B|A,C) \ne P(B|C)$, even though $P(A|B) = P(B)$, this is because both $A$ and $B$ increase the probability of $C$. Hospitalization, $C$, is the collider, being a common effect of $A$ and $B$.

user67724
  • 333