9

An exponential family distribution $p$ in the canonical form can be written as

$p(x|\theta) = h(x)\exp(\theta^\top T(x) - A(\theta))$

where $A(\theta)$ is the log partition function, $T(x)$ is the sufficient statistics, and $h(x)$ is the base measure (according to this Wikipedia page). For simplicity, let us consider a one dimensional $x$.

What is the restriction of the form of $h(x)$ ? My intuition tells me that $h(x)$ cannot be arbitrary because otherwise we can set $T(x)=0$, and leave all the "work" to $h(x)$.

For example, I understand that a Student's t is not in the exponential family. Let $t(x)$ be the density of a t distribution. If we set $T(x)=0$ and $h(x) = t(x)$, then

$A(\theta) = \log \int h(x)\exp(\theta^\top 0) \,dx = \log 1 = 0$,

and $p(x|\theta) = h(x) = t(x)$ implying that the t is in the exponential family.

What did I miss here ?

kjetil b halvorsen
  • 77,844
wij
  • 2,123

2 Answers2

5

Since $p(x|\theta)$ must integrate to 1, $h(x)$ must be non-negative, but that's the only restriction (according to page 111 in this book).

However, I think the question highlights a common confusion (at least one that I've had before). There isn't just one exponential family of distributions. Rather, there are many such exponential families as mentioned in the Exponential family Wikipedia article:

exponential families are in a sense very natural sets of distributions to consider.

The choice of the functions $h$ and $T$ specify the exponential family (i.e. model) and the parameter vector $\theta$ corresponds to a particular member (i.e. distribution) in that family.

Indeed, if you choose a some fixed degrees of freedom for the t-distribution (let's say $\nu = 3$), you could as you say let $T(x) = 0$ and $h(x) = t(x|\nu=3)$ which, following the formula on from the Student's t-distribution Wikipedia article, should give $$ h(x) = \frac{1} {\sqrt{3\pi}\,\Gamma(\frac{3}{2})} \left(1+\frac{x^2}{3} \right)^{-2}\!.$$

However, this doesn't give you the family of t-distributions, i.e. the set of functions $\{t(\cdot|\nu) : \nu > 0\}$. With this base distribution, you could construct a more interesting exponential family of distributions by using more interesting sufficient statistics $T(x)$, but you would not be able to design $T$ such that parameters $\theta$ would correspond to the $\nu$ parameter of the t-distribution.

teichert
  • 236
  • I'd also mention that, while we can "design" $T(x)$, it is defined by Fisher as "a statistic... with respect to... [$f(x; \theta$)] if no other statistic that can be calculated from the same sample provides any additional information as to the value of the parameter". Since forcing the T distribution into the exponential family form requires us to set $T(x)$ = 0, and $A(\theta)$ = 0, you're saying that no data provides information about $\theta$. In an exponential family parametrized by $\theta$, distributions from the family are unidentifiable. – Jake Stevens-Haas Oct 21 '18 at 21:57
  • @JakeStevens-Haas: the answer explains that an (infinity of) exponential family(ies) can be built out of this choice of function $h(x)$, which essentially modifies the dominating measure from the Lebesgue measure $\text d\mu$ to the Student measure $h(x)~\text d\mu$. Each of these exponential families is associated with a specific $T(x)$ in the exponential part and none corresponds to the Student's $t$ family. – Xi'an Jul 29 '21 at 03:45
1

The central restriction is that $h(x)$ cannot depend on the parameters $\theta$. Intuitively, the base measure is the distribution of $x$ when the exponential term is given no weight i.e. $\theta=0$. As mentioned in a comment to your question, this is related to the distinction between a family of distributions being an exponential family in itself vs. any single distribution being the member of some family.

Bonnevie
  • 41