1

The entropy of a random variable $X$ is defined as $\mathbb{E}(-\log(f(X)))$ (where $f$ is the pdf of X, https://en.wikipedia.org/wiki/Entropy_(information_theory)).

Is there any general relationship between the entropy of $X$ and the possibility to "more or less accurately" predict the value of $X$ using some constant $a$? That is, are there known general relationships between $\mathbb{E}(\log(X))$ and measures such as

$$ \min_a \mathbb{E}[|X-a|] \tag 1$$ $$ \min_a \mathbb{E}[(X-a)^2] \tag 2$$ $$ \min_a \mathbb{E}[I(X=a)] \tag 3$$ ?

For a $Be(p)$, for example, it seems that the two kinds of metrics (entropy on the one hand, and either of (1), (2), or (3) on the other) move jointly with $p$ (i.e., if $Be(p)$ has higher entropy than $Be(p')$, then $Be(p)$ is also less accurately predictable --- higher minimum expected error --- than $Be(p')$ in terms of (1), (2), or (3)).

Is something like that true more generally? For example, is it true in general that an increase in entropy implies a decrease in predictability as measured by (1), (2), or (3)? If it's not true in general, is true for some class of random variables larger than the class of $Be(p)$?

FZS
  • 395
  • I guess for (1), since $\min_a \mathbb{E}[(X-a)^2] = \mathbb{E}[(X-\mathbb{E}(X))^2] = Var(X)$, the question boils down to whether RV's with lower entropy systematically have lower variance. Intuitively, it feels like there should be counter-examples to this, though I still need to take the time to think about it and construct such a potential counter-example if one exists. – FZS Dec 17 '22 at 14:25
  • 1
    Caeteris paribus, the entropy gives some information about variance, whence (via Chebyshev's Inequality, among others) it is related to the standard error of the mean. The initial c.p. condition is crucial, though, because as a general proposition entropy tells you nothing about predictability. You have to constrain the space of possible distributions. – whuber Dec 17 '22 at 15:22
  • In part it depends on the definition of entropy being used. Shannon's entropy is borrowed from physics, where entropy is a measure of disorder. A cloud has higher entropy than an ice cube, since a cloud allows for many more ways to arrange water molecules than a cube’s crystalline structure does. In an analogous way, a random message has a high Shannon entropy — there are so many possibilities for how its information can be arranged — whereas one that obeys a strict pattern has low entropy. Given this definition, the lower the entropy, the more predictable are the events. – user78229 Dec 17 '22 at 18:31
  • @whuber: Thanks for the comment. What do you mean by c.p.? Would you be able to elaborate a little on "You have to constrain the space of possible distributions"? Are there famous classes of distributions for which entropy is "more directly" related to some "standard" measure of predictability? – FZS Dec 18 '22 at 02:38
  • 1
    See the first two words of my comment. – whuber Dec 18 '22 at 15:43

1 Answers1

1

I think there are counter-examples for all three measures. That is, for all three measures, there are cases where the entropy remains unchanged but the measure changes. Below are my proposed counter-examples.

Mean absolute distance (1)

Note that

$$\min_a \mathbb{E}[|X-a|] = \mathbb{E}[|X-\text{med}(X)|].$$

Let $f_Y(1) = f_Y(2) = f_Y(3) = 1/3$ and $f_Z(0) = f_Z(2) = f_Z(4) = 1/3$.

We have $\text{med}(Y) = \text{med}(Z) = 2$ and $H(Y) = H(Z) = -\log(1/3)$, though

$$ \mathbb{E}[|Z-\text{med}(Z)|] = 4/3 > 2/3 = \mathbb{E}[|Y-\text{med}(Y)|].$$

Mean squared distance (2)

Note that

$$\min_a \mathbb{E}[(X-a)^2] = \mathbb{E}[(X-\mathbb{E}(X))^2] = Var(X).$$

Let $Y = Be(p)$ and $Z = 2*Be(p)$. Note that the two RVs have the same entropy:

$$ \begin{align} H(Y) && = && f(0) * [-\log(f(0))] && + && f(1) * [-\log(1-f(1))] \\ &&=&& p* [-\log(p)] && + && (1-p)*[-\log(1-p)]. \end{align}$$ $$ \begin{align} H(Z) && = && f(0) * [-\log(f(0))] && + && f(2) * [-\log(1-f(2))]\\ && =&& p* [-\log(p)] && +&& (1-p)*[-\log(1-p)]. \end{align}$$

Of course, the two RVs have different variances with $Var(Z) = 4*Var(Y)$.

This seems to hold for any discrete RV, that is, if $Z = a + bY$, the two RVs have different variances ($Var(Z) = a^2 Var(Y)$), but have the same entropy.

As noted in the answer @whuber points at in the comments (How does entropy depend on location and scale?), this is not true for continuous distributions ("scaling a continuous variable [by $\sigma$] (which, for $\sigma \geq 1$ "stretches" or "smears" it out) increases its entropy by $\log(\sigma)$.").

Mean error rate (3)

Note that

$$\min_a \mathbb{E}[I(X=a)|] = \mathbb{E}[I(X=\text{mode}(X))].$$

Let $f_Y(1) = 1/8$, $f_Y(2) = 1/2$, $f_Y(3) = 3/8$ and $f_Z(1) = 1/4$, $f_Z(2) = 1/2$, $f_Z(3) = 1/4$.

We have $\mathbb{E}[I(Y=\text{mode}(Y))] = 1/2 = \mathbb{E}[I(Z=\text{mode}(Z))]$. However, $H(Y) \neq H(Z)$.

FZS
  • 395