1

The Variation of Information, $VI(X;Y)$, can be defined by the respective conditional entropies through the identity, $VI(X;Y) = H(X|Y)+H(Y|X)$.

I am curious as to what the relative weightings of the conditional entropies mean that make up the Variation of Information.

For example, if we let $w =\frac{H(Y|X)}{H(X|Y)}$, what would $w<1$ or $w>1$ mean with respect to the interaction of information between $X$ and $Y$? Would such a 'measure' serve any relevant purpose?

kjetil b halvorsen
  • 77,844
Mari153
  • 880
  • Since entropies are already on a logarithmic scale, taking ratios ought to make little sense. In fact, if these are ("differential") entropies of continuous distribution functions, the ratio might not even be defined and can be negative or positive. – whuber May 31 '22 at 13:50
  • @whuber - so, in effect, you are saying that the components that make up a verified information measure, don't mean anything? Now that is an interesting interpretation – Mari153 May 31 '22 at 21:21
  • Of course not. So let me repeat: computing this ratio is meaningless. It's analogous to taking the ratio of a winter high temperature of 5 degrees C and winter low of -5 degrees C and trying to interpret the resulting value of -1. – whuber May 31 '22 at 21:26
  • @whuber - a further point. Curious as to why your comment refers to continuous distribution rather than a discrete distribution? – Mari153 May 31 '22 at 21:26
  • The two kinds of entropies are quite different. The differential entropy will change when you change the unit of measurement, for instance (exactly like changing from Centigrade to Fahrenheit for temperature measurements), but that cannot occur with discrete distributions. See https://stats.stackexchange.com/questions/415435. – whuber May 31 '22 at 21:28
  • @whuber - so if the ratio is meaningless, then is the absolute difference also meaningless? – Mari153 May 31 '22 at 21:29
  • Differences of entropies are meaningful. After all, they correspond to ratios of commensurate quantities before the logarithms were applied. – whuber May 31 '22 at 21:51

0 Answers0