A mathematical quantity designed to measure the amount of randomness of a random variable.
Entropy is a mathematical quantity designed to quantify the uncertainty about the occurrence of outcomes of a random variable. It is expressed as a function of the outcome probabilities of the random variable. Any measure for entropy must satisfy a few conditions:
- Continuity: The function must be continuous in all its arguments.
- Maximum: The function should have a maximum when all outcome are equally probable.
- Symmetry: The function must remain unchanged under a switch of arguments.
A commonly adopted measure is the Shannon entropy, $\mathrm{H}(p_1,p_2,\ldots,p_n)$ (when $p_1,p_2,\ldots p_n$ are the $n$ outcome probabilities of a random variable $X$). This measure is defined as follows:
$$\mathrm{H}(p_1,p_2,\ldots,p_n) = -\sum_{i=1}^n p_i \log p_i$$
