We know that for discrete variables \begin{equation} D(p(x),q(x))=\mathbb{E}_{p}\left(\log\frac{p(x)}{q(x)}\right) \end{equation} where $p(x)$ and $q(x)$ are probability mass functions. Can this be extended to \begin{equation} D(p(x,y,z),q(x,y,z))=\mathbb{E}_{p}\left(\log\frac{p(x,y,z)}{q(x,y,z)}\right) \end{equation} ? where now $p(x,y,z)$ and $q(x,y,z)$ are joint probability mass functions. Is the notation correct?
Asked
Active
Viewed 408 times
1
-
3Yes, that's correct. The way you've written KL divergence using expectations is valid for any pair of distributions defined on the same space (not limited to discrete values; also works for reals, vectors, etc.). Notation looks fine to me, as long as you specify that $D(\cdot, \cdot)$ means KL divergence. Everyone writes things a little bit differently. – user20160 Nov 27 '19 at 08:26
-
Yes, and see here for an explanation and interpretation of that expectation. – kjetil b halvorsen Nov 27 '19 at 10:57
-
@user20160: do you want to post your comment(s) as an answer? Better to have a short answer than no answer at all. Anyone who has a better answer can post it. – Sycorax Feb 17 '21 at 06:57
1 Answers
1
Yes, that's correct. The way you've written KL divergence using expectations is valid for any pair of distributions defined on the same space (not limited to discrete values; also works for reals, vectors, etc.). Notation looks fine to me, as long as you specify that (⋅,⋅) means KL divergence. Everyone writes things a little bit differently. – user20160
I've created this answer from a comment so that this question is answered. Better to have a short answer than no answer at all.
Sycorax
- 90,934