Distribution of a conditional expectation

Question

I am interested in calculating the distribution of a conditional expectation, I would like some help figuring out if I am on the right track.

Consider a variable $x\sim f(x)$ and a noisy signal $s$ of $x$ drawn from $g(s;x)$. Consider $E(x|s)$ computed using Bayes' rule. I am interested in the distribution of all possible values of $E(x|s)$ which I will call $q(z)$, which I think I can define as $$ q(z) = \int 1(E(x|s)=z) g(s;x)ds $$ where 1 is the indicator function. My question is whether my definition of $q(z)$ makes sense or if there is a better/conventional way to define it. I am somewhat confused because the law of iterated expectations states that $E_sE(x|s)=E(x)$ but this should only imply $E(q(z))=E(f(x))$, correct?

To help understanding what I am after, for example, consider $x\sim U[0,1]$ and $s$ one Bernoulli draw with parameter $p=x$. Then using Bayes' rule $E(x|s=1)=2/3$ and $E(x|s=0)=1/3$ (the prior is a Beta(1,1), the two posteriors are Beta(2,1) and Beta(1,2) with averages 1/3 and 2/3). The two values of the conditional expectations which are equally likely because of the symmetry of f(x), therefore
$$q(x) = \begin{cases} 1/2 & \text{if } x=1/3 \\ 1/2 & \text{if } x=2/3 \\ 0 & \text{otherwise} \end{cases}$$

Answer 1

Possibly this question may help: Bayes' Theorem Intuition

In my answer there I used the following figure:

Bayes rule can be viewed as taking a slice out of the joint distribution and regarding the conditional distribution.

Obviously the mean of $P(B)$ and $P(B|A)$ are not the same. But we can express $E(B)$ by summing/integrating over all the expectations of $E(B|A)$ (the expectation of $B$ given $A$).

---

You could relate the two as follows

$$E(B) = \int_{\forall a} E(B|A) f_A(a) \text{d}a \tag{1}$$

And with

$$E(B|A) = \frac{\int_{\forall b} b f(b,a) \text{d}b}{\int_{\forall b} f(b,a) \text{d}b}$$

and

$$f_A(a) = \int_{\forall b} f(a,b) \text{d}b$$

if you substitute them into (1) you get

$$\begin{array}{}E(B)& = &\int_{\forall a} \frac{\int_{\forall b} b f(a,b) \text{d}b}{\left[\int_{\forall b} f(a,b) \text{d}b\right]} \left[\int_{\forall b} f(a,b) \text{d}b \right]\text{d}a\\& = &\int_{\forall a} \int_{\forall b} b f(a,b) \text{d}b\text{d}a \end{array} $$

---

If you want to define something like the distribution of $E(B|A)$ then you could consider the transformation rules for probability distributions which scales the density with the derivate of the transformation (as well as sums over a potential multiple cases of $a$ that match the same mean):

$$f_{E(B|A)}(e) = \sum_{\forall a:E(B|A)=e} f_{A}(a) \left|\frac{\text{d}a}{\text{d}e}\right|$$

This might be what you were trying to do with the integral including the indicator function.

Answer 2

I would work through this problem as follows:

$$\langle x| s \rangle = \int x \cdot p(x|s) dx$$

I think this is different to what you've done, you seem to be integrating against s but any expression for $\langle x| s \rangle$ must have explicit dependence on s so you can't have s dependence integrated out

Then you can use Bayes to write this as

$$\langle x| s \rangle = \int x \cdot \frac{p(s|x)p(x)}{p(s)} dx$$

where I just used $p$ to generically denote "the pdf of", as per the usual Bayes' theorem notation. By your definitions, $p(x)=f(x)$ and $p(s|x)=g(x;s)$

Also, by the chain rule of probability,

$p(s) = \int p(s|x)p(x)dx$ so putting this all together

$$\langle x| s \rangle = \frac{\int x \cdot g(x;s)f(x)dx}{\int g(x;s)f(x)dx}$$

In your example, $f(x)$ is a uniform between $[0,1]$ and $g(x;s)=x^{s}(1-x)^{1-s}$, and thus

$$\langle x|s\rangle = \frac{\int _{0}^{1}x^{s+1}(1-x)^{1-s}}{\int _{0}^{1}x^{s}(1-x)^{1-s}}=\frac{\beta(s+2, 2-s)}{\beta(s+1, 2-s)}$$

(where $\beta$ denotes the beta function)

this simplifies to $\frac{s+1}{3}.$

Note that this gives you the same values you got, but I think you're somehow confusing x and s. s can take the values 0 or 1. If the coin flip comes up 0, chances are its probability of coming up heads (the result of your prior draw from the uniform) was lower than 0, and indeed the expectation is 1/3 and conversely it's 2/3 if the coin flip comes up 1. Note that $\langle x|s \rangle$ is not a function of x, as it's an expectation value of the random variable x, all x-dependence has been integrated out. It explicitly depends on s, and coincidentally in this example, there are two possible values of s and if you sum the values for $s=0$ and $s=1$ you get 1, but this is a coincidence. This is not a distribution over s, it isn't (generally speaking, albeit in this case it is) normalised wrt s.

So I think you're broadly on the right track but just need to make sure you're not confusing x and s. One thing that I don't think helped is that you've used this notation $g(x;s)$ to denote $p(s|x)$, which really I think you should be denoting as $g(s;x)$, or ideally $p(s|x)$. The notation $g(s;x)$ to me implies a pdf/pmf over s, with parameters x (and thus normalised wrt s). In your simple example, this made sense as $x$ was literally the parameter value, but more generally it might be the case that $s$ is distributed according to parameters $\theta$, and $\theta$ is actually a function of x, in which case the distribution of s still depends on x but isn't directly parametrised through it, hence me saying using $p(s|x)$ is probably more helpful notation in terms of keeping track of what's going on.

Answer 3

I am answering my own question, thanking @gazza89 for putting me on the right track. Given $()=\int(|)() $, and $E(x|s)$ defined by Bayes' rule, the distribution of the conditional expectations $q(E(x|s))$ can be computed as follows.

If $E(x|s)$ is an invertible function of s, calling $E^{-1}(z)$ its inverse, then q(z) is obtained by a change of variables, for the discrete $s$ case $q(z) = p(E^{-1}(z))$, and for the continuous case $q(z) = p(E^{-1}(z))\left | \frac{dE^{-1}(z)}{dz}\right|$

If $E(x|s)$ is not invertible $$ q(z) = \int _{E(x|s)=z}p(s)ds $$

where $$ is the indicator function i.e. we need to sum up/integrate $p(s)$ for all values of s such that $E(x|s)=z$.