Measuring the "distance" between two multivariate distributions

Question

I'm looking for some good terminology to describe what I'm trying to do, to make it easier to look for resources.

So, say I have two clusters of points A and B, each associated to two values, X and Y, and I want to measure the "distance" between A and B - i.e. how likely is it that they were sampled from the same distribution (I can assume that the distributions are normal). For example, if X and Y are correlated in A but not in B, the distributions are different.

Intuitively, I would get the covariance matrix of A, and then look at how likely each point in B is to fit in there, and vice-versa (probably using someting like Mahalanobis distance).

But that is a bit "ad-hoc", and there is probably a more rigorous way of describing this (of course, in practice I have more than two datasets with more than two variables - I'm trying to identify which of my datasets are outliers).

Thanks!

Dunno why, but a Mantel test flashed in front of my eyes when I read your post. — Roman Luštrik, Nov 06 '10 at 15:43

score 19 · Answer 1 · answered Oct 28 '10 at 13:47

19

Hmm, the Bhattacharyya distance seems to be what I'm looking for, though the Hellinger distance works too.

answered Oct 28 '10 at 13:47

Emile

1,097

1

you mention Bhattacharyya and Helling then accept an answer speaking about KL... At the end what was your choice and why? – Simon C. Jan 23 '19 at 16:20
2

I believe it was KL divergence, but ... that was in 2010 and my memory is far from perfect. – Emile Jan 23 '19 at 17:21
ahah yes I guessed that, but thank you anyway! – Simon C. Jan 23 '19 at 17:40

Gavin Simpson · Accepted Answer · 2010-10-28T14:27:09.980

18

There is also the Kullback-Leibler divergence, which is related to the Hellinger Distance you mention above.

edited Oct 28 '10 at 14:27

answered Oct 28 '10 at 14:19

Gavin Simpson

47,626

3

can one calculate the Kullback-Leibler divergence of points without making an assumption of the underlying probability density the points came from ? – Andre Holzner Nov 06 '10 at 16:22

skyde · Answer 3 · 2011-07-01T02:47:22.257

14

Heuristic

Minkowski-form
Weighted-Mean-Variance (WMV)

Nonparametric test statistics

2 (Chi Square)
Kolmogorov-Smirnov (KS)
Cramer/von Mises (CvM)

Information-theory divergences

Kullback-Liebler (KL)
Jensen–Shannon divergence (metric)
Jeffrey-divergence (numerically stable and symmetric)

Ground distance measures

Histogram intersection
Quadratic form (QF)
Earth Movers Distance (EMD)

edited Jul 01 '11 at 02:47

answered Jul 01 '11 at 02:39

skyde

465

score 10 · Answer 4 · edited Nov 06 '10 at 15:10

10

The most complete survey is provided in Statistical Inference Based on Divergence Measures by Leandro Pardo, Complutense University, Chapman Hall 2006.

edited Nov 06 '10 at 15:10

whuber

322,774

answered Nov 06 '10 at 11:07

Mark Salmon

131

score 1 · Answer 5 · answered Jul 12 '18 at 20:26

Few more measures of "Statistical Difference"

Permutation test (by Fisher)
Central Limit Theorem & Slutsky’s theorem
Mann-Whitney-Wilcoxin test
Anderson–Darling test
Shapiro–Wilk test
Hosmer–Lemeshow test
Kuiper's test
kernelized Stein discrepancy
Jaccard similarity
Also, hierarchical clustering deals with similarity measures between groups. The most popular measures of group similarity are perhaps the single linkage, complete linkage, and average linkage.

Measuring the "distance" between two multivariate distributions

5 Answers5

Linked