4

I am trying to understand the example described in the WGAN paper about learning parallel lines with various divergences. More specifically the setup is as follows:

Let $Z \sim [0, 1]$ the uniform distribution on the unit interval. Let $\mathbb{P}_0$ be the distribution of $(0, Z) \in \mathbb{R}^2$, uniform on a straight vertical line passing through the origin (meaning $g_{0}(z) = (0, z)$). Now let $g_{\theta}(z) = (\theta, z)$ with $\theta$ a single real parameter. The authors claim that:

  • $KL(\mathbb{P}_0||\mathbb{P}_{\theta}) = KL(\mathbb{P}_{\theta}||\mathbb{P}_0)$ = $\infty$ if $\theta \neq 0$ and $0$ otherwise
  • $JS(\mathbb{P}_0||\mathbb{P}_{\theta})$ = $\log2$ if $\theta \neq 0$ and $0$ otherwise

Intuitively, the results make sense but trying to reproduce the calculations I am a bit confused about how are we going to use the KL/JS divergence in this setting. For KL/JS divergences we have:

$KL(\mathbb{P}_0||\mathbb{P}_{\theta})=\int_{0}^{1} (g_0(z) \log\frac{g_0(z)}{g_{\theta}(z)}) \mathrm{d}z $

$KL(\mathbb{P}_{\theta}||\mathbb{P}_{0})=\int_{0}^{1} (g_{\theta}(z) \log\frac{g_{\theta}(z)}{g_{0}(z)}) \mathrm{d}z$

$JS(\mathbb{P}_0||\mathbb{P}_{\theta}) = \frac{1}{2}\int_{0}^{1} (g_{\theta}(z) \log\frac{g_{\theta}(z)}{\frac{g_{0}(z)+g_{\theta}(z)}{2}})\mathrm{d}z + \frac{1}{2}\int_{0}^{1} (g_{0}(z) \log\frac{g_{0}(z)}{\frac{g_{0}(z)+g_{\theta}(z)}{2}})\mathrm{d}z $

Can anyone help me on how to incorporate the single point information ($0$ or $\theta$) in these integrals and do a bit more analytically the calculations? Do we have to take a separate integral over a single point because of the definition of $g_{0}(z), g_{\theta}(z)$?

dkoutsou
  • 43
  • You forgot about taking density in the integrals – Jakub Bartczuk Apr 16 '18 at 09:39
  • Thank you, I edited my answer to include densities. – dkoutsou Apr 16 '18 at 09:44
  • You still use $g_{\theta}$ etc which are not densities, and the formulas still don't make sense (for example you divide two 2D vectors). – Jakub Bartczuk Apr 16 '18 at 09:50
  • You are right that $g_{0}, g_{\theta}$ are vectors that contain random variables and not densities. But that is my point of confusion. How are we going to incorporate the vector information in the KL divergence definition? If I just put the density definitions, I will end up with $0$ everywhere since I don't take into account the $0, \theta$ positioning. – dkoutsou Apr 16 '18 at 09:56

1 Answers1

1

Notice that the two distributions can be written as

$\newcommand{\Pb}{\mathbb{P}}$ $$ \Pb_0 = 1_{\{x_1 = 0\}} 1_{\{0 \leq x_2 \leq 1\}} \frac{1}{x_2} \\ \Pb_{\theta} = 1_{\{x_1 = \theta\}} 1_{\{0 \leq x_2 \leq 1\}} \frac{1}{x_2} \\ $$

so when you are computing the KL divergence, you are computing

\begin{align*} D_{\mathrm{KL}}(\Pb_0 \| \Pb_{\theta}) &= \int_{\mathcal{X}} \mathbb{P}_0(x) \log \frac{\Pb_{\theta}(x)}{\Pb_0(x)} \text{d}x \\ &= \int_{0}^1 \frac{1}{x_2} \log \frac{1_{\{x_1 = \theta\}}\frac{1}{x_2}} {1_{\{x_1 = 0\}}{\frac{1}{x_2}}} \text{d}x_2 \end{align*} When $\theta \neq 0$, the two densities have no common support, hence the KL divergence is $\infty$. When $\theta = 0$, we get

$$ D_{\mathrm{KL}}(\Pb_0 \| \Pb_{\theta}) = \int_0^1 \frac{1}{x_2} \log \frac{1_{\{x_1 = 0\}} \frac{1}{x_2}}{1_{\{x_1 = 0\}}\frac{1}{x_2}} \text{d} x_2 = \int_0^1 \frac{1}{x_2} \log(1) \text{d} x_2 = 0 $$

The same apply when computing $D_{\mathrm{KL}}(\Pb_{\theta} \| \Pb_0)$.

VHarisop
  • 358