6

Suppose I am adjusting distribution $Q$ to make get a best fit for distribution $P$, should I minimize $KL(P||Q)$ or $KL(Q||P)$? What is the difference?

Related question:

Intuition on the Kullback-Leibler (KL) Divergence

Haitao Du
  • 36,852
  • 25
  • 145
  • 242
  • It is not a distance measure. It is not a metric. – Carl Sep 17 '17 at 04:06
  • 1
    Can you clarify how your question differs from the linked thread? If it isn't what you want, please edit your question to state what you learned & what you still need to know. Then we can provide the information you need without just duplicating material elsewhere that already didn't help you. – gung - Reinstate Monica Sep 18 '17 at 14:39

2 Answers2

4

You usually want $KL(Q||P)$. That's from $P$ to $Q$. I remember that it goes from right to left, just like the notation for conditional probabilities.

You want the expectation being taken with respect to the true distribution $P$. That way, sample averages can be assumed to converge to the true expectations, by the law of large numbers.

Taylor
  • 20,630
  • thanks for the answer. What do you think about this book page 74? – Haitao Du Sep 18 '17 at 16:44
  • 1
    @hxd1011 oh that's interesting. It makes sense that doing it different ways gives you different answers (I see all of these comments on asymmetry above). I would definitely be interested in seeing some real-life examples of people going the other way with it. – Taylor Sep 18 '17 at 16:50
3

The KL divergence is not a distance this is why the alternative word divergence is used instead. If you want symmetry you can take the sum of $KL(Q||P)$ and $KL(P||Q)$ as mentioned in one of the answers in the linked post.

The intuition is that you do not know the true distribution $Q$ so you make an estimate or a guess of the true distribution $P$. The two may be in the same parametric family or they may not be similar at all. Therefore to understand some notion of how far your assigned probabilities (a view of the events) are from the true probabilities (how much the two perspectives diverge) you would take an expectation under your estimated probabilities. However, you would still like this divergence to be $0$ if you somehow specified the exact model, to ensure this you subtract that true value. Using the logarithm property $log(x/y) =log(x) - log(y)$ then gives you the KL-divergence equation.

Lucas Roberts
  • 4,259