Aside from the exponential family, where else can conjugate priors come from?

Question

Do all conjugate priors have to come from the exponential family? If not, what other families are known to have/produce conjugate priors?

Answer 1

As explained for example in Section 3.3.3 of book "The Bayesian choice" by Christian Robert, there is indeed a narrow connection between exponential families and conjugate priors, but there are conjugate priors available for certain non-exponential families. He calls these "quasi-exponential", however, because they are families for which sufficient statistics of finite dimension not increasing with sample size do exist.

Here is an example for the uniform distribution, whose support depends on the parameter of the distribution and can therefore not be an exponential family (as is well-known):

Here, the Pareto distribution is a conjugate prior for the parameter $b$ of the uniform distribution on $[0,b]$.

The density of the Pareto distribution with parameters $c>0$ and $% \alpha >0$ is \begin{equation*} f(x)=\alpha c^{\alpha }x^{-\alpha -1} \end{equation*} for $x\geq c$ and $f(x)=0$ else.

The prior of the parameter $b$ of a uniform distribution on $[0,b]$ is a Pareto distribution with $c_{0}$ and $\alpha _{0}$, \begin{eqnarray*} \pi (b) &=& \left\{ \begin{array}{ll} \alpha _{0}c_{0}^{\alpha _{0}}b ^{-\alpha _{0}-1} & \quad \text{if } b \geq c_{0} \\ 0 & \quad \text{else.}% \end{array} \right. \\ &\propto &\left\{ \begin{array}{ll} b ^{-\alpha _{0}-1} & \quad \text{if }b \geq c_{0} \\ 0 & \quad \text{else.}% \end{array}% \right. \end{eqnarray*} The likelihood for the data $y_{1},\ldots ,y_{n}$, given $b$, is \begin{equation*} f\left( y |b\right) = \left\{ \begin{array}{ll} \prod_{i=1}^{n}\frac{1}{b }=b ^{-n} & \quad \text{if }0\leq y_{i}\leq b \text{ for all }i=1,\ldots ,n \\ 0 & \quad \text{else.}% \end{array}% \right. \end{equation*} The product of likelihood and prior is the non-normalized posterior \begin{eqnarray*} \pi \left( b |y\right) &\propto &\pi (b )f\left( y |b\right) \\ &=&\left\{ \begin{array}{ll} \alpha _{0}c_{0}^{\alpha _{0}}b ^{-\alpha _{0}-1}b ^{-n} & \quad \text{if }b \geq c_{0}\text{ and }0\leq y_{i}\leq b \text{ for all }i=1,\ldots ,n \\ 0 & \quad \text{else.}% \end{array}% \right. \\ &\propto &\left\{ \begin{array}{ll} b ^{-\alpha _{0}-n-1} & \quad \text{if }b \geq c_{0}\text{ and }% 0\leq y_{i}\leq b \text{ for all }i=1,\ldots ,n \\ 0 & \quad \text{else.}% \end{array}% \right. \\ &\propto &\left\{ \begin{array}{ll} b ^{-\alpha _{1}-1} & \quad \text{if }b \geq c_{1} \\ 0 & \quad \text{else.}% \end{array}% \right. \end{eqnarray*} with \begin{eqnarray*} \alpha _{1} &=&\alpha _{0}+n \\ c_{1} &=&\max\bigl(\max_{i}y_{i},c_0\bigr). \end{eqnarray*} Hence, the posterior is Pareto distributed.