Intuitively, why is cross entropy a measure of distance of two probability distributions?

Question

For two discrete distributions $p$ and $q$, cross entropy is defined as

$$H(p,q)=-\sum_x p(x)\log q(x).$$

I wonder why this would be an intuitive measure of distance between two probability distributions?

I see that $H(p,p)$ is the entropy of $p$, which measures "surprise" of $p$. $H(p,q)$ is the measure that partly replaces $p$ by $q$. I still do not understand the intuitive meaning behind the definition.

Answer 1

Minimizing the cross entropy is often used as a learning objective in generative models where p is the true distribution and q is the learned distribution.

The cross entropy of p and q is equal to the entropy of p plus the KL divergence between p and q.

$H(p, q) = H(p) + D_{KL}(p||q)$

You can think of $H(p)$ as a constant because $p$ comes directly from the training data and is not learned by the model. So, only the KL divergence term is important. The motivation for KL divergence as a distance between probability distributions is that it tells you how many bits of information are gained by using the distribution p instead of the approximation q.

Note that KL divergence isn't a proper distance metric. For one thing, it is not symmetric in p and q. If you need a distance metric for probability distributions you will have to use something else. But, if you are using the word "distance" informally then you can use KL divergence.

Answer 2

Agree with the top answer that cross-entropy, as a distribution dissimilarity measure, may be more narrowly applicable to situations when we are comparing an estimated probability distribution (q) against the true probability distribution (p).

If we only consider the true probability distribution p, its entropy is the expected value of the corresponding log-probability (also could be viewed as the amount of "self-information": That is, the number of bits required to encode the information): $$E_{p}(log_2(p)) = -\Sigma_{i=1}^{n}p_{i}log_{2}(p_{i})$$

When computing cross-entropy between the estimated distribution q and true p, we replace the log-probability $log_2(p)$ with the estimated counterpart $log_2(q)$. After re-arranging the equation we can the following:

\begin{equation} \begin{aligned} E_{p}(log_2(q)) &= -\Sigma_{i=1}^{n}p_{i}log_{2}(q_{i})\\ &= -(\Sigma_{i=1}^{n}p_{i}log_{2}(\frac{q_{i}}{p_{i}}p_{i}))\\ &= -\Sigma_{i=1}^{n}p_{i}log_{2}(p_{i}) - \Sigma_{i=1}^{n}p_{i}log_{2}(\frac{q_{i}}{p_{i}})\\ &= entropy(p) + KLdivergence(p||q) \end{aligned} \end{equation}

In this way, the extra KL divergence term is a relative entropy measure that describes how many extra bits we would need when replacing the true distribution p with the estimated distribution q. The asymmetry is also visible (as mentioned in other answers above): $$-\Sigma_{i=1}^{n}p_{i}log_{2}(\frac{q_{i}}{p_{i}}) \neq -\Sigma_{i=1}^{n}q_{i}log_{2}(\frac{p_{i}}{q_{i}})$$

In the context of machine learning, cross-entropy is a commonly used loss function and by minimizing it we learn the model parameters.

When coming to comparing two distributions in a broader sense, you might be looking for metrics such as:

1. The asymmetric Kullback-Leibler divergence: scipy.special.kl_div
2. The symmetric Jensen-Shannon divergence: scipy.special.distance.jensenshannon

Answer 3

I think cross entropy cannot be used as a distance since H(p,q) is not H(q,p). It is a uni-directional measure, so it can be used to measure the loss (or difference) from the original probability like KD divergence.