1

According to the Wiki, Jensen–Shannon divergence (JSD) is bounded between [0,1]. I am having trouble understanding why this is.

Let's say $p_1 ~ N(\mu_1,\sigma^2)$, and $p_2 ~ N(\mu_2,\sigma^2)$ (same standard deviation). Then $JSD(p_1||p_2) = \frac{1}{4}((\frac{\mu_1-(\frac{\mu_1+\mu_2}{2})^2}{\sigma^2})+(\frac{\mu_2-(\frac{\mu_1+\mu_2}{2})^2}{\sigma^2}))$. I would expect that in the limit as $\sigma \rightarrow 0, JSD \rightarrow \infty$.

However, while the above is a problem for Kullback–Leibler divergence (KLD), it is not a problem for JSD according to this post. So, what am I doing wrong? How is JSD bound even in the above case?

Edit:

I got to this JSD formula by the following. $\mu_{avg} = \frac{1}{2}(\mu_1+\mu_2)$. I assumed that the mixture distribution would have the same standard deviation $\sigma$ (maybe that is wrong).

By using the KLD formula here. With means $\mu_1$, $\mu_{avg}$ and shared standard deviation $\sigma$, we get KLD1 = $\frac{(\mu_1-\mu_{avg})^2}{2\sigma^2}$, KLD2 = $\frac{(\mu_2-\mu_{avg})^2}{2\sigma^2}$.

Then I used JSD = $\frac{1}{2}KLD1 + \frac{1}{2}KLD2$, which gave me my formula above. I take it that somewhere in here I went wrong?

Richard Hardy
  • 67,272
lowlyprogrammer
  • 111
  • 3
    Could you explain how you arrive at this formula for the JS divergence of two Normal distributions? – whuber Oct 12 '23 at 18:55
  • @whuber I edited my answer above to show my work. I'm guessing that is where I went wrong? – lowlyprogrammer Oct 12 '23 at 19:13
  • 2
    You can find a correct formula at https://stats.stackexchange.com/questions/16608. https://stats.stackexchange.com/questions/416204 has some illustrations. – whuber Oct 12 '23 at 20:29

1 Answers1

2

For the general case: The fact that the Jensen–Shannon divergence is bounded by 1 is proven in the following paper:

Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory, 37(1), 145-151.

More specifically, inequality (4.3) states that

$$ JSD(p_1||p_2) \leq -\pi_1 \log \pi_1 - \pi_2 \log \pi_2 $$

for any $\pi_1$ and $\pi_2$ so that $\pi_1 \leq 0$, $\pi_2 \leq 0$, and $\pi_1 + \pi_2 =1$. The right-hand side of the inequality is a binary entropy function and its maximum is 1 and is reached for $\pi_1 = \pi_2 =0.5$, hence the upper bound.

The lower bound simply comes from the positivity of the Kullback-Leibler divergence.

Camille Gontier
  • 2,616
  • 6
  • 13