How can I prove that KL-divergence is not symmetric?

Question

https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence

Thanks a lot!

Thanks for the remark. I did that. Still couldn't find any mathematical proof. — user309678, Jan 30 '21 at 04:58
Hint: One example suffices! So calculate one example and see. You can find one example in https://stats.stackexchange.com/questions/188903/intuition-on-the-kullback-leibler-kl-divergence/189758#189758 — kjetil b halvorsen, Jan 30 '21 at 05:26

score 1 · Answer 1 · answered Jan 30 '21 at 05:33

1

It would suffice to show that with a single example. One possibility is as follows:

Define $$ P(x) = \begin{cases} 1, \text{with probability } 0.5 \\ -1, \text{with probability } 0.5 \end{cases} $$ and $$ Q(x) = \begin{cases} 1, \text{with probability } 0.1 \\ -1, \text{with probability } 0.9 \end{cases} $$

You can easily verify that $D_{KL}(P||Q) = 0.5*\ln(0.5/0.1) + 0.5*\ln(0.5/0.9)$ and $D_{KL}(Q||P) = 0.1*\ln(0.1/0.5) + 0.9*\ln(0.9/0.5)$ and they are not equal.

answered Jan 30 '21 at 05:33

Tom Bennett

728

Is that also applies to the continuous space? – user309678 Jan 30 '21 at 05:35
Yep. You can make an example along the same lines. – Tom Bennett Jan 30 '21 at 05:35
I get lost because of the integral instead of summation... – user309678 Jan 30 '21 at 05:36
Try two normal distributions with different standard deviations. – Tom Bennett Jan 30 '21 at 05:37
Could you please elaborate? – user309678 Jan 30 '21 at 05:37
You can let P be a normal distribution with mean 0 and standard deviation 1, Q be another normal distribution with mean 0 and standard deviation 2, and compute the KL distance for P||Q and Q||P.. – Tom Bennett Jan 30 '21 at 05:41
Yeah, I meant I'm not sure how to plug this into the formula... Can you please tell me? Thanks! – user309678 Jan 30 '21 at 05:46
Try to look at https://en.wikipedia.org/wiki/Normal_distribution and plug the density with $\sigma=1, \mu=0$ as P and same density with $\sigma=2, \mu=0$ as Q to compute KL(P||Q). Then swap the roles of P and Q to compute KL(Q||P). – Tom Bennett Jan 30 '21 at 05:56

How can I prove that KL-divergence is not symmetric?

1 Answers1