2

I've recently started learning about Bayesian statistics, and I came across this very nice answer by Xi'an https://stats.stackexchange.com/a/129908/268693, which [in my slight paraphrasing] says the following: Given a family of distributions $\{f(\cdot|\theta): \theta \in \Theta \}$ defined on a sample space $\mathcal{X}$, and a prior distribution $\pi$, we require that

$$ \int_{\Theta} f(x|\theta) \pi(\theta) \,d\theta < \infty \quad \text{ for all } x \in \mathcal{X}; $$

otherwise, we do not obtain a valid prior distribution $\pi(\theta)$, and so Bayesian inference is not possible. This leads me to the following question:

What are some families of distributions $\{f(\cdot|\theta): \theta \in \Theta \}$ that one might encounter in practice for which there exists a set $E \subset \mathcal{X}$ of positive measure such that $$ \int_{\Theta} f(x|\theta) \,d\theta = + \infty \quad \text{ for all } x \in E? $$

In other words, I'm curious if there is a family of distributions for which a uniform prior leads to an "improper posterior" such that the problem cannot be remedied by re-defining the $f(\cdot|\theta)$'s on a set of measure zero.

Here are a couple of examples I came up with:

  • The Cauchy distribution: $f(x|\theta) = \frac{1}{\theta \pi [1 + (x/\theta)^2 ]}, \; x > 0, \; \theta > 0$. In this case, $\int_{0}^{\infty} f(x|\theta) \,d\theta = + \infty$ for all $x > 0$.

  • A rather contrived example: For each $\theta \in \Theta := (1,\infty)$, let $f(x|\theta) := \frac{1}{\theta^x}$ for each $x \in \mathcal{X} := (0,\infty)$. Then $$ \int_{\Theta} f(x|\theta) \,d\theta := \begin{cases} \frac{1}{x-1} & \text{ if } x > 1 \\ \infty & \text{ if } 0 < x \leq 1 \end{cases} $$

(Would $f(x|\theta) = 1/\theta^x$ ever be used in practice?) Are there any other such examples? I am especially interested in examples such as last one, where the set $\{x \in \mathcal{X}: \int_{\Theta} f(x|\theta) \,d\theta = + \infty \}$ has positive and finite measure.

Leonidas
  • 121
  • 8
  • 1
    Just to clarify: are you asking for which type of likelihood function $p(x|\theta)$ an improper prior (i.e., $\int_{\Theta} p(\theta) d\theta = \infty$) will result in an improper posterior (i.e., $\int_{\Theta} p(\theta|x) d\theta = \infty$)? As mentioned here, this is the case when the likelihood is flat. – Durden Apr 09 '23 at 23:13
  • The question is problematic in that the answer depends on the parameterisation chosen for the model $f(\cdot|\theta)$, since there is no preferential parameterisation for which the Uniform prior would be more justified than for others... – Xi'an Apr 10 '23 at 08:01
  • @Durden: Thanks for the link, I read through the thread. When you say "when the likelihood is flat", do you mean something like $f(x|\theta) = \theta x^{\theta - 1}, , 0 \leq x \leq 1, , \theta > 0$ in the case where $x = 0$ ? Do you know of any other non-pathological examples that one might actually encounter in practice? – Leonidas Apr 10 '23 at 13:39
  • @Leonidas the likelihood being 'flat' generally hints at identification issues (here's one example from the archives; I'm sure there are others): the likelihood as constructed considers a wide range of parameter values as (nearly or perfectly) equally likely, and as a result your posterior will just be your prior. This problem has many names across disciplines, but you'll find the most active recent discussion under the keyword 'posterior collapse' in ML. – Durden Apr 10 '23 at 17:47

0 Answers0