0

I'm struggling with a form of viewing Shannon Entropy. Cover & Thomas say that entropy is the expected value of information content. So there is a random variable $X$ with distribution $P(X)$ and its entropy would be given by $H(X) = \underset{x\in X}{\sum}p(x)\log_2\frac{1}{p(x)}$.

My problem is when calculating entropy of a series of outcomes of $X$. If these outcomes are $x_1, x_2\in X$, is it appropriate to calculate the entropy of this set as $H(x_1, x_2) = \frac{1}{2}\log_2\frac{1}{p(x_1)} + \frac{1}{2}\log_2\frac{1}{p(x_2)}$? If this is not Shannon Entropy, then is it another known measurement? Also, how can I calculate the entropy of such set?

Nick Cox
  • 56,404
  • 8
  • 127
  • 185
blandre
  • 1
  • 1
    It is appropriate if and only if $p(x_1) = p(x_2) = 0.5$. If not, it's something else, namely the average of $\log_2 p_i$. – Nick Cox Apr 03 '19 at 18:41
  • Thanks! $p(x_1)$ and $p(x_2)$ are not $0.5$. But $\log_2\frac{1}{p(x)}$ is the information content, and multiplying by $0.5$ would make for the expected value of information content in this case, would it not?

    Edit: I also think it is not Shannon Entropy, but I'm struggling with the fact that within a set of occurrences probabilities will change, and how it affects entropy is still not very clear to me.

     – blandre Apr 03 '19 at 18:45
  • 1
    No, because differing probabilities enter the definition. If I work out the expected height of a population I don't just throw possible heights in and average them. I have to weight by their probabilities. – Nick Cox Apr 03 '19 at 18:50
  • Related: Statistical interpretation of Maximum Entropy Distribution – kjetil b halvorsen Jul 12 '22 at 02:12

0 Answers0