Are there better measures of entropy

Question

Related question here

I am trying to measure the uniformity of multimodal distributions and am looking into using entropy.

I would like a measure of entropy that is higher for the first distribution than the second.

However, Shannon entropy is order independent (invariant under permutations) so the above distributions have the same Shannon entropy.

Is there anyway to capture clumping in the measure of entropy?

My thinking is that under the heat equation, it should take longer for the second distribution's Shannon entropy to become asymptotic so it should have lower entropy.

---

Let there be 6 bins evenly filled out of 12. Shannon entropy doesn't distinguish between the case A where all even bins are filled vs case B where bins [0,3) and [9, 12) are filled

index	0	1	2	3	4	5	6	7	8	9	10	11
A	0	1/6	0	1/6	0	1/6	0	1/6	0	1/6	0	1/6
B	1/6	1/6	1/6	0	0	0	0	0	0	1/6	1/6	1/6

---

Right--but so what? The indexes of the bins are irrelevant. If they are relevant to you, then you do not need entropy; you need another concept. But in that case, what are you trying to characterize? @whuber

I though this might have been studied and you could tell me?

Answer 1

Shannon's entropy distinguishes between those two scenarios:

Two bins:

$$- \sum_{i=1}^2 \frac{1}{2} \log \frac{1}{2} = \log 2$$

Ten bins:

$$- \sum_{i=1}^{10} \frac{1}{10} \log \frac{1}{10} = \log 10$$

Finite $n$ bins:

$$- \sum_{i=1}^{n} \frac{1}{n} \log \frac{1}{n} = \log n$$

And note that $\log 10 > \log 2$. And since logarithms are monotonic functions, we know that more bins means a bigger number. I.e. $m > n \implies \log m > \log n$.

---

You are right that Shannon's entropy is invariant to the permutations of the bins. If the entropy depended on permutation, then it would distinguish between different versions of your two examples. You have ten bins in the first example. Such a permutation-varying entropy would consider the order of bins (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) different from (1, 2, 3, 4, 5, 6, 7, 8, 10, 9), and yet different from (1, 2, 3, 4, 5, 6, 7, 9, 8, 10),.... for all $10! = 3628800$ permutations of ten bins.

---

Re: 10/20 bins filled example:

You might be interested in normalizing the entropy to the max possible entropy of 20 bins.

Let us assume there is 20 outcomes, then for any uniform choice on ten of those bins we will have a normalized entropy of:

$$0 \leq \frac{H_{10}}{H_{\max}} = \frac{\log 10}{ \log 20} = \log_{20}(10) \approx 0.768621786840241 \leq 1$$

---

Is there anyway to capture clumping in the measure of entropy?

I'm not sure this is really what you want, but you could compute a sort of weighted entropy:

$$H_{w(x)} = - \sum_x [w(x)p(x)] \log [w(x)p(x)]$$

where $0 \leq w(x) \leq 1$ is some kind of weight based on the distances between bins which satisfies $\sum_x w(x) = 1$.

Answer 2

Variance can be generalized to be about any point rather than just the mean (i.e. how spread out is the distribution from this point) $$\mu \mapsto\sum p_i (x_i - \mu)^2.$$

We can also "invert" the variance to get a similarity measure for each point (how bunched is the distribution about this point) $$\mu \mapsto \sum p_i e^{-(x_i - \mu)^2}.$$

Can then take the log of this to get a surprise function $$\sigma :=\mu \mapsto -\log\sum p_i e^{-(x_i - \mu)^2}.$$

Entropy is then given by the formula $$\prod \sigma(x_i)^{-p_i}$$ as explained by blog post 1 and 2 given by @kjetil answer. These links do a very good job of developing the language you need to think about entropy.

• Abundance: the proportions in which the species occur (e.g. 50% grass, 30% clover, 20% daisies)
• Similarity: the extent to which the species are related (e.g. an ecosystem made up of 50% snails and 50% slugs should probably be regarded as less diverse than one made up of 50% snails and 50% bats).

Shannon entropy is only a good measure when working with discrete metric spaces i.e. the distance between different "bins" is infininte. However, our bins have a natural distance between them (i.e. the difference of the indices) and so we need something more