10

Consider the Bayesian posterior $\theta\mid X$. Asymptotically, its maximum occurs at the MLE estimate $\hat \theta$, which just maximizes the likelihood $\operatorname{argmin}_\theta\, f_\theta(X)$.

All these concepts—Bayesian priors, maximizing the likelihood—sound super principled and not at all arbitrary. There’s not a log in sight.

Yet MLE minimizes the KL divergence between the real distribution $\tilde f$ and $f_\theta(x)$, i.e., it minimizes

$$ KL(\tilde f \parallel f_\theta) = \int_{-\infty}^{+\infty} \tilde f(x) \left[ \log \tilde f(x) - \log f_\theta(x) \right] \, dx $$

Woah—where did these logs come from? Why KL divergence in particular?

Why, for example, does minimizing a different divergence not correspond to the super principled and motivated concepts of Bayesian posteriors and maximizing likelihood above?

There seems to be something special about KL divergence and/or logs in this context. Of course, we can throw our hands in the air and say that’s just how the math is. But I suspect there might be some deeper intuition or connections to uncover.

Yatharth Agarwal
  • 389
  • 1
    You can find some ideas here: https://stats.stackexchange.com/questions/188903/intuition-on-the-kullback-leibler-kl-divergence/189758#189758 – kjetil b halvorsen Feb 13 '19 at 21:49
  • @kjetilbhalvorsen The previous title sounded like a duplicate; I apologise. I’ve made an edit, and it should be clear why this question is not a duplicate. – Yatharth Agarwal Feb 15 '19 at 01:48
  • The other questions asks, “What is KL divergence, and why isn’t it symmetric?” The answers explain the concept of a divergence, and some info about KL. In contrast, this question asks “Why does the Bayesian posterior concentrate around the minimiser of KL divergence?” Simply explaining how divergences don’t have to be symmetric and explaining KL and stating KL is connected to MLE fails to address the crux of the question here: why among the many possible divergence does KL in particular have a special connection to the Bayesian posterior. Does this make sense? – Yatharth Agarwal Feb 15 '19 at 01:50
  • Yes, it makes sense, but there is still a problem. The posterior depends also on the prior, and if that is strong, the posteriorcan have a max away from the mle. But the prior is absent from your question. – kjetil b halvorsen Feb 15 '19 at 08:26
  • @kjetilbhalversen I meant asymptotically with more and more IID samples and under the (stringent) conditions under which the prior does not matter asymptotically! – Yatharth Agarwal Feb 15 '19 at 20:01
  • Tjen YouTube should make an exit to Therese Q starting tjat – kjetil b halvorsen Feb 15 '19 at 20:11
  • Can you explain your question more? (I.e. the Bayesian posterior minimizes KL divergence from what?) Seems like your explanation says that the MLE $\hat{\theta}$ is the parameter such that $f_\hat{\theta} = \arg\min_{\tilde{f}} KL(\tilde{f} || f_{\hat{\theta}})$ which is vacuously true. Is this what you mean or am I missing something? – lady gaga Oct 01 '21 at 20:19
  • I've seen Bayes' rule written as an optimization (for example in variational inference) where the posterior $p$ is defined as $p = \arg\min_q KL(q||p)$ which is vacuously true. And it is written this way because $KL( \cdot|| p)$ can be nicely rewritten in terms of the prior and likelihood. – lady gaga Oct 01 '21 at 20:20

1 Answers1

5

Use of logarithms in calculations like this comes from information theory. In the particular case of the KL divergence, the measure can be interpreted as the relative information of two distributions:

$$\begin{equation} \begin{aligned} KL(\tilde{f} \parallel f_\theta) &= \int \limits_{-\infty}^\infty \tilde{f}(x) (\log \tilde{f}(x) - \log f_\theta (x)) \ dx \\[6pt] &= \Bigg( \underbrace{- \int \limits_{-\infty}^\infty \tilde{f}(x) \log f_\theta(x) \ dx}_{H(\tilde{f}, f_\theta)} \Bigg) - \Bigg( \underbrace{- \int \limits_{-\infty}^\infty \tilde{f}(x) \log \tilde{f}(x) \ dx}_{H(\tilde{f})} \Bigg), \\[6pt] \end{aligned} \end{equation}$$

where $H(\tilde{f})$ is the entropy of $\tilde{f}$ and $H(\tilde{f}, f_\theta)$ is the cross-entropy of the $\tilde{f}$ and $f_\theta$. Entropy can be regarded as measures of the average rate of produced by a density (thought cross-entropy is a bit more complicated). Minimising the KL divergence for a fixed value $\tilde{f}$ (as in the problem you mention) is equivalent to minimising the cross-entropy, and so this optimisation can be given an information-theoretic interpretation.

It is not possible for me to give a good account of information theory, and the properties of information measures, in a short post. However, I would recommend having a look at the field, as it has close connections to statistics. Many statistical measures involving integrals and sums over logarithms of densities are simple combinations of standard information measures used in measure theory, and in such cases, they can be given interpretations in terms of the underlying levels of information in various densities, etc.

Michael Hardy
  • 9,818
Ben
  • 124,856
  • Looking into information theory sounds promising! Thanks for pointing me to it. – Yatharth Agarwal Feb 15 '19 at 01:54
  • Obviously, you can’t explain an entire mathematical field in a StackExchange post, but would you have any particular references to they the log comes up? – Yatharth Agarwal Feb 15 '19 at 01:55
  • I just think there’s such deep intuition behind why, say, e is in Euler’s equation and such, that there’s similar intuition lurking here. Maybe a product somewhere makes the natural logarithm arise. I’m not sure. – Yatharth Agarwal Feb 15 '19 at 01:56
  • @Yatharth the logarithm arises here because of its central role in the definition of Shannon entropy. As for "why" a logarithm is appropriate for a measure of information, as opposed to another function, take a look at theorem 2 in Shannon's "Mathematical Theory of Communication". Also, Jayne's "Information Theory and Statistical Mechanics" is a nice introduction. – Nate Pope Feb 19 '19 at 21:18