3

I'm confused by the phrasing I've seen about exponential families. What does it mean to say "an" exponential family. Why not "the" exponential family?

From a pdf from Berkely: "we define an exponential family of probability distributions as those distributions..."

From Statistical Inference (George Casella, Roger L. Berger): "Let $X_1, X_2, ..., X_n$ be iid observations from a pdf $f(x|\theta)$ that belongs to an exponential family..."

I've also ready things that say, "this distribution belongs to an exponential family..."

But what are the different exponential families? Why not just say the exponential family? If there are multiple exponential families, why haven't I ever seen something like, "the binomial distribution belongs to exponential family A, while this other distribution belongs to family B..."?

I've searched around, and can't find a list of these families. How can there be an exponential family if there are not more than one of them?

kjetil b halvorsen
  • 77,844
Nick Koprowicz
  • 1,078
  • 2
    In these contexts, an individual exponential family is something like a normal, Poisson, gamma, etc. I was initially a bit confused about this as well... – WavesWashSands Mar 02 '20 at 17:56
  • 3
    "Exponential family" is like "sports car", a specific class with many specific examples, yet clear distinctions from others. – AdamO Mar 02 '20 at 18:22
  • We have many posts about this: see https://stats.stackexchange.com/search?q=%22exponential+family%22 – whuber Mar 02 '20 at 20:23
  • 2
    Thanks for your responses. I think the missing piece for me was that a distribution itself could be a family. I just honestly didn't understand the idea of a "family" at all; but now I understand it's just a set of pdf's with a certain form with varying parameters. So binomial(n, p) is an example of an exponential family, and binomial(100, 0.5) is a member of that family? – Nick Koprowicz Mar 03 '20 at 01:49
  • 2
    This shouldn't be closed in my opinion - it's a common point to get stuck on, and it's not exactly the topic any of the linked questions. (The accepted answer to the first linked question does answer it though.) – N. Virgo Mar 14 '21 at 10:33
  • 1
    For the sake of having a concrete answer in the comments at least: a "family of distributions" is probably best thought of as a function that takes the parameters as input and returns a distribution as output. So indeed, the binomial family is an example of an exponential family, and binomial(100, 0.5) is indeed a member of that family. The phrase "the exponential family" usually refers to the set of all exponential families. It's a bit of an unfortunate term though, because it makes the word "family" mean two different things at the same time, so I prefer to avoid it. – N. Virgo Mar 14 '21 at 10:33
  • 1
    My reason to reopen this is because it is more specifically about the nomenclature and use of 'an exponential family' versus 'the exponential family'. It occurs in discussions here and here. The suggested duplicates do not provide an answer about the terminology. – Sextus Empiricus Mar 26 '24 at 15:13
  • 1
    @N.Virgo The meaning of a family of distributions is fulsomely explained in my post at https://stats.stackexchange.com/a/63461/919. – whuber Mar 26 '24 at 15:28
  • @Sextus Isn't my explanation at the beginning of https://stats.stackexchange.com/a/519715/919 sufficiently clear? (There are many others like this but that's the first I found...) – whuber Mar 26 '24 at 15:28
  • 1
    @whuber all the densities of the form $f(x,\theta) = \exp(\eta(\theta)T(x) + A(\theta)+B(x))$ create 'the exponential family'. But is a subset 'a exponential family'? The definition is clear, but some historical background about the convention would be interesting. – Sextus Empiricus Mar 26 '24 at 15:33
  • So $f(x,\theta)$ defines a family, but is it 'a member of the exponential family' or 'an exponential family'. – Sextus Empiricus Mar 26 '24 at 15:50
  • @Sextus "The" is a misnomer. Each exponential family is determined (essentially) by $T.$ One has to be careful in any event not to confuse this with a family of exponential distributions (that is, $\Gamma(1)$ and its location-scale relatives). Thus, context and clarity matter and ought to supersede any effort to declare there's a universal, conventional terminology. – whuber Mar 26 '24 at 16:23

1 Answers1

2

Possibly a reason for the confusing terminology is that 'the exponential family' is a family of families $f(x|\theta)$ *.

Why the name 'exponential'? Each of the families has the property that it can be factorized, and with the factor that is a function of the sufficient statistic $T(X)$ and the parameters $\theta$ being an exponential term.

$$f(x|\theta) \propto h(x) \cdot e^{T(x)\cdot \eta(\theta)}$$

  1. So we can consider the family consisting of all families $f(x|\theta)_{h,T,\eta}$.
  2. And we can consider a single specific member family $f(x|\theta)_{h,T,\eta}$.

Because they are both families, the term 'exponential' can get stuck to both of them. It is typical to call the first the exponential family, and the second an exponential family. (and exponential becomes an adjective, like in the use 'a discrete distribution family').

* if we would consider 'the exponential family' as a family of distributions $f(x)$, then it would encompass the entire space of possible distribution functions. Every possible probability distribution occurs as a specific distribution in a family of the exponential family. For every potential distribution $f(x)$ there are some $h,T,η,θ$ that match. What is relevant to describe a member of the exponential distribution family is the parameterisation. For example, we can have distribution families that, due to their parameterisation, are not in the exponential family.

Scortchi - Reinstate Monica
  • 29,907
Sextus Empiricus
  • 77,915
  • Personally I am not fan of the use of 'an exponential distribution' but it is extremely common and used by almost everyone. But in the same way I don't like to write something like 'X is normally distributed' and prefer 'X is normal distributed', so what I prefer doesn't mean that it is what is convention. – Sextus Empiricus Mar 26 '24 at 16:29
  • I wonder how the uses of these two uses have historically developed. – Sextus Empiricus Mar 26 '24 at 16:31
  • The term family, referring to a family of distributions, tends to be introduced only in more careful/theoretical discussions. Elsewhere "The [X] distribution" can designate either the family of all [X] distributions ("the normal distribution is parametrized by its mean & variance") or all members of that family taken individually ("the normal distribution is symmetric"). (Cf "the field vole is common in England" & "the field vole lives in shallow burrows".) ... – Scortchi - Reinstate Monica Mar 27 '24 at 10:01
  • .... So (here goes my theory), for many, "exponential families" or "an exponential family" will be their first encounter with family. Naturally enough, they misconstrue the adjective as pertaining to a collection of families of distributions rather than to each family of distributions in the collection, & come up with "the exponential family" (perhaps helped along by the required distinction from "the exponential distribution") . Needless to say, the misnomer propagates. – Scortchi - Reinstate Monica Mar 27 '24 at 10:01
  • @Scortchi-ReinstateMonica by calling 'the exponential family' a misnomer, you mean that the set of exponential families can not be a family? – Sextus Empiricus Mar 27 '24 at 10:31
  • Sorry: no, not at all. Just that the adjective "exponential" seems misplaced. On the other hand you can call a family of tall people "a tall family", so perhaps it's rather subjective. But I do suspect that "the exponential family" is a term coined rather accidently. – Scortchi - Reinstate Monica Mar 27 '24 at 10:51
  • 1
    An early occurence of the term exponential is Girshick and Savage 1951 "Bayes and Minimax Estimates for Quadratic Loss Functions", and they use the term mostly as "an exponential family", and "exponential families", using the adjective for the members, but in footnote they speak of "...they deal with several specific cases of the exponential family of distributions." and use the adjective for the family of families. So this accident already happened in one of the first uses of the term 'exponential family'.... – Sextus Empiricus Mar 27 '24 at 11:16
  • ... Later in the 50's one can see different uses by different writers. It depends on whether they phrase some theory or property like "for the distributions in the exponential family, we have that..." (e.g. Kendall often uses this approach) versus "for an exponential distribution, we have that" (e.g. Lehman often uses this approach) Before in the 40's we see similar split in referring to a single function of a certain property, versus a function from the set/class of functions with a certain property. – Sextus Empiricus Mar 27 '24 at 11:19