Conditional Probability vs Conditional Probability Distribution

Question

I am having hard time interpreting the relationship, if any, between conditional probabilities vs. conditional probability distributions, in particular, regarding the number of random variables required to be able to define these two concepts.

As I understand, conditional probability can be defined for events involving a single random variable. For example, we could ask what is the probability that a die roll resulted in 4 given that the number is larger than 3, i.e., $P(X=4 | X>3)$.

However, the conditional probability distribution is based on the concept of joint distribution, which requires two or more random variables.

My question is whether it makes sense to talk about the conditional probability distribution of a single random variable or, conversely, whether a conditional probability statement like the one above can be related to a conditional probability distribution?

Answer 1

A conditional probability "distribution" is essentially just a bunch of conditional probabilities, sufficient to fully characterise the conditional behaviour of one random variable given another event or random variable. A probability "distribution" can be characterised in various different ways (e.g., by a probability measure, mass/density function, CDF, generating function, etc.) and while there is no single mathematical object that is the distribution, we may refer to them as such as a shorthand.

In general, there are two main classes of mathematical objects which would characterise a "conditional probability distribution" and which we might refer to as such:

• Conditional distribution at a given conditioning point: This is characterised by any function that fully characterises the probabilitic behaviour of $X$ conditional on a specific event $Y=y$.

• Conditional distribution for any conditioning point: This is characterised by any function that fully characterises the probabilitic behaviour of $X$ conditional on any value of another random variable $Y$.

Let me illustrate this by example. Suppose we have two random variables $X$ and $Y$ and suppose we define the conditional cumulative distribution function (CDF):

$$F(x|y) \equiv \mathbb{P}(X \leqslant x | Y=y) \quad \quad \quad \text{for all } x \in \mathscr{X} \text{ and } y \in \mathscr{Y}.$$

The function $F( \cdot |y)$ for a fixed value of $y$ fully characterises the distribution of $X$ given the conditioning point $Y=y$, so we would consider this to be a "conditional distribution" in the shorthand sense previously described. The function $F( \cdot | \cdot)$ fully characterises the distribution of $X$ given any conditioning point for $Y$, so we would also consider this to be a "conditional distribution", again, in the shorthand sense. (The latter object is much more general, and it actually gives a whole bunch of conditional distributions, corresponding to each of the possible values for $Y=y$.)

Answer 2

It seems to me we can use what we already know, provided we have heard of distribution functions and conditional probabilities. Thus, the following remarks offer nothing new, but I hope that in making them the basic simplicity and familiarity of the situation will become apparent.

---

When you have any real-valued random variable $X$ and an event $\mathcal E$ (defined on the same probability space, of course), then you can extend the definition of a (cumulative) distribution function in the most natural way possible: namely, for any number $x,$ define

$$F_X(x;\mathcal E) = \Pr(X\le x\mid \mathcal E).$$

When $\mathcal E$ has positive probability you can even avoid all technicalities and apply the elementary formula for conditional probability,

$$\Pr(X\le x\mid \mathcal E) = \frac{\Pr(X\le x\,\cap\,\mathcal E)}{\Pr(\mathcal E)}.$$

The numerator, which might look strange to the mathematically sophisticated reader, is the probability of the intersection of two events. The conventional shorthand "$X\le x$" stands for the set of outcomes where $X$ does not exceed $x:$ $\{\omega\in\Omega\mid X(\omega)\le x\}.$

This extends the usual distribution function very nicely in the sense that when $\Omega$ is the universal event (that is, the underlying set of all outcomes in the probability space), then since $(X\le x)\subseteq \Omega$ and $\Pr(\Omega)=1,$

$$F_X(x)= \Pr(X\le x) = \frac{\Pr(X\le x\,\cap\,\Omega)}{1} = \frac{\Pr(X\le x\,\cap\,\Omega)}{\Pr(\Omega)}= F_X(x;\Omega) .$$

---

Comments

Note that only one random variable $X$ is needed, showing that the concept of conditional distribution does not depend on a joint distribution. As a simple example, the right-truncated Normal distribution studied at Expected value of x in a normal distribution, GIVEN that it is below a certain value is determined by a Normally-distributed random variable $X$ and the event $X\le T$ (for the fixed truncation limit $T$).

Another example, just to make these distinctions very clear, models a population of people where we are interested in their sex and age (at a specified time, because both these properties can change!). By agreeing on a unit of measure of age (seconds, say), and (for simplicity) focusing on those people with a definite sex, we may take the sample space to be

$$\Omega = \{\text{male}, \text{female}\}\times [0,\infty).$$

Elements of $\Omega$ represent people. A sample from $\Omega$ could be represented by rows in a two-column table: one for sex, the other for age. That's what the Cartesian product $\times$ in the definition of $\Omega$ means.

The probabilities of interest will attach to intervals of ages for each sex separately (or combined). Thus, relevant events will be composed out of age intervals of the form $\{\text{male}\}\times (a,b]$ (for lower and upper ages $a$ and $b$ of males) and $\{\text{female}\}\times (a,b]$ (an interval of ages for females).

As a shorthand, "$\{\text{male}\}$" is the event $\{\text{male}\}\times [0,\infty) = \{(\text{male},x)\mid x \ge 0\},$ and similarly for "$\{\text{female}\}.$" By definition, these are both events -- or "subpopulations" if you like.

Let $X$ be the random variable giving the age of a person rounded to the nearest year. Then (for instance) we might be interested in $F_X$ (the distribution of all ages), of $F_X(\ \mid \{\text{male}\})$ (the distribution of male ages), or of $F_X(\ \mid \{\text{female}\}).$

This nice example shows that the conditioning event $\mathcal E$ (the sex) needn't have anything to do with $X$ (the age).

Clearly, this formulation of conditional distributions does not require us to define a random variable to condition on a characteristic like sex in the example. We could have done it that way, and there are some analytical and computational advantages to doing so, but conceptually such a construct would be artificial and superfluous.

When there are multiple random variables $(X,Y)$ (and $Y$ can be vector-valued), nothing new emerges because conditioning on $Y$ means conditioning on the events it defines.

Answer 3

Well I think the term "conditional probability" or "conditional probability distribution" can both extend to two or more variables (correct me if I am wrong). For example, let us suppose we have $X,Y,Z$ are i.i.d random variables uniformly distributed on (0,1) for simplicity. Then we are required to find the conditional probability of $P\left(X \ge YZ|Y>\frac{1}{2}\right)$. This should be a valid question about finding conditional probability.

Then,

\begin{align}P\left(X \ge YZ|Y>\frac{1}{2}\right) &= \frac{P\left(X \ge YZ,Y>\frac{1}{2}\right)}{P(Y>\frac{1}{2})} \\&= \frac{\int_0^1\int_\frac{1}{2}^1\int_{yz}^1 1 dxdydz}{\frac{1}{2}}\\&= 2 \int_0^1\int_\frac{1}{2}^1(1-yz)dydz \\&= 2\int_0^1 \left(\frac{1}{2}-\frac{3z}{8}\right)dz\\&=2 \left( \frac{1}{2}-\frac{3}{16}\right)\\&=\frac{5}{8}\end{align}

Similarly, $$P\left(X \ge YZ|Y \le\frac{1}{2}\right)= \frac{\int_0^1\int_0^\frac{1}{2}\int_{yz}^1 1 dxdydz}{\frac{1}{2}} = \frac{7}{8},$$ as you can verify.

In a nutshell, the concept of conditional probability is not only valid on single variable case.