A measure of the difference between two probability distributions for a given random variable or set of events.
In information theory, the cross-entropy between two probability distributions $p$ and $q$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution $q$, rather than the true distribution $p$.
The cross-entropy of the distribution $q$ relative to a distribution $p$ over a given set is defined as follows: $$ H(p,q)=-\mathbb{E}_p(\log q) $$ where $\mathbb{E}_p(\cdot)$ is the expected value operator with respect to the distribution $p$.
Source: Wikipedia.
Excerpt source: Brownlee "A Gentle Introduction to Cross-Entropy for Machine Learning" (2019).
