In a Bayesian context, is the data random or given?

Question

Let us recall Bayes Theorem, where $\mathcal{D}$ is the data and $\theta$ is our parameter(s) of interest.

$ \operatorname{p}(\theta|\mathcal{D}) = \frac{{\operatorname{p}(\mathcal{D}, \theta)}}{\operatorname{p}(\mathcal{D})} = \frac{{\operatorname{p}(\mathcal{D}|\theta)}{\operatorname{p}(\theta)}}{\operatorname{p}(\mathcal{D})} $

We then have:

• The conditional probability $\operatorname{p}(\theta|\mathcal{D})$. As a conditional probability, it is only a function of $\theta$, and $\mathcal{D}$ is assumed to be given.
• The joint probability $\operatorname{p}(\mathcal{D}, \theta)$. As a joint probability, it is a function of two random variables, thus $\mathcal{D}$ is not given, but random.
• The marginal probability $\operatorname{p}(\mathcal{D})$. As a marginal probability, it is a function of a random variable, thus $\mathcal{D}$ is not given, but random.

It seems that $\mathcal{D}$ has a dual-role. In the left side of the equation it is a constant, whereas in the right hand side it is a (vector of) random variable(s). Is the posterior a function of $\theta$ and $\mathcal{D}$ (as in the right-hand side) or is the posterior just a function of $\theta$ (as in the left-hand side)?

---

Auxiliary Questions

1. I have omitted the case of the likelihood $\operatorname{p}(\mathcal{D}|\theta)$, but maybe it is at the core of what is happening. The likelihood is a function of $\theta$, once the data has been observed (https://stats.stackexchange.com/a/138707/180158). However, it is also used as a probability density function of the data given $\theta$. Otherwise, we would not use the chain rule to factorize the joint distribution $\operatorname{p}(\mathcal{D}, \theta)$ into the conditional and marginal distributions $\operatorname{p}(\mathcal{D}|\theta)\operatorname{p}(\theta)$. Does this dual-behavior depend on the presence of the other terms in Bayes theorem?

2. In practice we tend to work only with the unnormalized (log) posterior (e.g. in Stan) : $\operatorname{p}(\theta|\mathcal{D}) \propto \operatorname{p}(\theta, \mathcal{D})$. Does the use of the unnormalized (log) posterior change anything regarding our terminology or assumptions of what $\mathcal{D}$ is?

Answer 1

In the beginning, that is, at the Bayesian modelling stage, there are two random entities, $\mathcal D$ and $\theta$, with a joint distribution with density $p(\theta,\mathcal D)$. Then comes the observation of the data, which is a realisation $d$ of the random variable $\mathcal D$. The random variable $\theta$ remains random since it is not observed but the realisation $d$ brings some degree of information about $\theta$, which is one considers its distribution conditional on the realisation $d$ of $\mathcal D$, with density $p(\theta|\mathcal D=d)$.

Note that in standard statistical modelling, the data is usually considered as the observed realisation of the random variable, rather than as a random variable. To quote Wikipedia,

It is assumed that there is a "true" probability distribution induced by the process that generates the observed data.

Now, when considering Bayes' theorem, $$\operatorname{p}(\theta|\mathcal{D}) = \frac{{\operatorname{p}(\mathcal{D}, \theta)}}{\operatorname{p}(\mathcal{D})} = \frac{{\operatorname{p}(\mathcal{D}|\theta)}{\operatorname{p}(\theta)}}{\operatorname{p}(\mathcal{D})}$$ it is a mere mathematical (functional) identity linking the five density functions involved in it and should better be written $$\operatorname{p}(\theta|d) = \frac{{\operatorname{p}(d, \theta)}}{\operatorname{p}(d)} = \frac{{\operatorname{p}(d|\theta)}{\operatorname{p}(\theta)}}{\operatorname{p}(d)}\qquad\forall\,\theta,d$$ as it holds for all possible entries $\theta,d$.

From a mathematical perspective, $\operatorname{p}(\theta|d)$ is a function of both $\theta$ and $d$ since changing the value of $\theta$ or the value of $d$ modifies (in general) the value of $\operatorname{p}(\theta|d)$. The same remark applies to the likelihood function. What may be confusing to the OP is the use of the notation $d$ in this paragraph as a generic data realisation, varying in a sample space, $\mathfrak D$ say, as opposed to the actual observed data realisation also denoted $d$ in the first paragraph. Which is why the observed data is sometimes denoted otherwise, $d^o$ for instance. With such notations, $\operatorname{p}(\theta|d)$ is a function of both $\theta$ and $d$, while $\operatorname{p}(\theta|d^o)$ is a function of $\theta$ only.