3

I'm trying to fashion survey experiments for which the demographic composition of the samples needs to be as close as possible to a 'golden' composition. For concreteness, let's say the 'golden' composition is 14% middle class, 35% upper-middle class, and 51% upper class.

I have a whole bunch of samples, each with slightly different compositions. For example:

S1 = 10% middle class, 41% upper-middle class, 49% upper class
S2 = 33% middle class, 33% upper-middle class, 34% upper class

I'm trying to find a good statistical measure to describe how close the distribution in each sample is to the ideal, 'golden' composition. Is there a simple metric here that I could use?

ttnphns
  • 57,480
  • 49
  • 284
  • 501
user139014
  • 133

1 Answers1

5

What about Kullback–Leibler divergence? This is a good measure (in terms of usability in the classification based machine learning) of differences between two probability distributions. This is an asymmetric measure, but as you are comparing many compositions with one golden standard it seems quite a good choice.

$$D_{KL}(P||Q) = \sum_i \ln \left ( \frac{P(i)}{Q(i)} \right ) P(i) $$

It is the expectation of the logarithmic difference between the probabilities P and Q, where the expectation is taken using the probabilities P.

It can be also treated as a difference in the entropy

$$D_{KL}(P||Q) = H(P,Q) - H(P)$$

where $H(P,Q)$ is a cross entropy and $H(P)$ is entropy.

COOLSerdash
  • 30,198
lejlot
  • 4,361
  • To confirm, it is okay to use K-L divergence with categorical distributions, correct? For example, if I had another golden standard like 55% from New Jersey, 5% from California, 40% from New York? – user139014 Aug 12 '13 at 07:59
  • 1
    Yes, this is often used measure for eg. computing how sure is ensamble of models about their multi class prediction (each gives its own $P(y|x)$ for categorical $y$'s, and $D_{KL}$ is used to measure if they reached a consensus. – lejlot Aug 12 '13 at 08:02