Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.
Mathematically a distance, $d$, or metric, is a function that satisfies the following properties. For two points $x, y, z$:
- $d(x,y) \geq 0$
- $d(x,y) = 0 \implies x = y$
- $d(x,y) = d(y,x)$
- $d(x,z) \leq d(x,y) + d(y,z)$
Note that certain concepts of distance in probability theory do not satisfy these properties. In particular, the KL-distance between two distributions is not symmetric, and doesn't satisfy the third property above.
Euclidean distance, Manhattan distance and Hamming distance are all common metrics.
