Two-sample Kolmogorov-Smirnov discrete distributions

Question

Let's say that two Census takers interview a number of couples and ask for the number of children they have. I want to compare the two resulting samples to find out if they could have come from the same population (let's say, the same country) or if they are more likely to have been drawn from two different populations.

I have been searching for quite some time and still could not find a definitive answer. Most sources state that the two-sample Kolmogorov-Smirnov test could be used if these were continuous distributions, which is not the case.

It seems that the two-sample chi-squared test, or the Wilcoxon-Mann-Whitney rank sum test could be used for discrete distributions, but these seem to suffer in the presence of many ties, which would be the case (most couples will have 0-3 kids).

Some have suggested the Wald statistic, which seems to be applicable to Poisson distributions. However, I don't assume that my problem follows any particular distribution.

In summary, is there a most appropriate test to find out if these two discrete/count samples come from the same population?

Answer 1

Some software implementations of the K-S test do not work at all well in case of ties, which I would expect to see in a survey counting numbers of children in individual families. Also, a two-sample Wilcoxon Rank Sum test may work poorly if you have many ties and small sample sizes.

However, a chi-squared test could be useful in the circumstances you describe. Maybe you have counts for 0-5 children in two surveys, each of 1000 families, as follows.

tx
[1] 359 387 187  48  12   7
ty
[1] 599 311  75  14   1   0

Then make a $2 \times 6$ table of counts:

TAB = rbind(tx,ty); TAB
   [,1] [,2] [,3] [,4] [,5] [,6]
tx  359  387  187   48   12    7
ty  599  311   75   14    1    0

Use a chisq.test in R to see if the two populations are the same.

chisq.test(TAB)
    Pearson's Chi-squared test


data:  TAB
X-squared = 151.23, df = 5, p-value < 2.2e-16
Warning message:
In chisq.test(TAB) : 
 Chi-squared approximation may be incorrect

The warning message appears because some of the expected counts in the computation of the chi-squared statistic are smaller than $5.$

 chisq.test(TAB)$exp
   [,1] [,2] [,3] [,4] [,5] [,6]
tx  479  349  131   31  6.5  3.5
ty  479  349  131   31  6.5  3.5

You could combine the last two columns to avoid these small expected values.

Alternatively, using chisq.test, as implemented in R. you could use parameter sim=T to simulate a more useful P-value near $0,$ so we reject $H_0$ that the two surveys sampled from the same population.

chisq.test(TAB, sim=T)
    Pearson's Chi-squared test 
    with simulated p-value 
    (based on 2000 replicates)


data:  TAB
X-squared = 151.23, df = NA, p-value = 0.0004998---------

---

Notes on other tests: (a) If you have one sample of size 20 from $\mathsf{Binom(10, .4}$ and another from $\mathsf{Binom}(10, .6),$ then the K-S test fails to find a difference between the two sample, gives an error message about ties, and there is no obvious cure for the ties.

    Two-sample Kolmogorov-Smirnov test
data:  a and b
D = 0.35, p-value = 0.1725
alternative hypothesis: two-sided
Warning message:
In ks.test(a, b) : cannot compute exact p-value with ties

(b) Similarly, a two-sample Wilcoxon test warns about ties, finds a significant difference, but sample sizes are too small to ignore the warning.

wilcox.test(a,b)
    Wilcoxon rank sum test 
    with continuity correction


data:  a and b
W = 106, p-value = 0.01015

alternative hypothesis: 
 true location shift is not equal to 0
Warning message:
In wilcox.test.default(a, b) : 
 cannot compute exact p-value with ties

With samples of size 200: [For large samples, R's implementation of this test uses a normal approximation that is more forgiving of ties, and no warning message is given.]

wilcox.test(rbinom(200,10,.4),rbinom(200,10,.6))
Wilcoxon rank sum test with continuity correction


data:  rbinom(200, 10, 0.4) and rbinom(200, 10, 0.6)
W = 5957, p-value < 2.2e-16
alternative hypothesis: 
 true location shift is not equal to 0

Answer 2

You can certainly use a two-sample Kolmogorov-Smirnov test with a discrete distribution. But it is possible that the typical asymptotic approximation used in many implementations will potentially require a much larger sample than would be needed with a continuous distribution. Regardless, the simple solution to that problem is to find an implementation that is randomization based or uses some other approximation for the p-value.

If you are using R, I believe the packages twosamples, KSgeneral, and dgof all appear to have KS test implementations that should work. I recall several others, but I can't find them at the moment.

---

ETA: I don't recommend immediately jumping to Chi-sq -- though it will also work in these situations. This is because Chi-sq doesn't have any understanding of ordering in the data. If one observation moves from 0 kids to 5 kids between samples, chi-sq will only see 1 observation moving, whereas KS will see 1 observation making a big jump. Having said that, non-permutation KS tests are notoriously underpowered, so Chi-sq may often be a good option -- even though it gives up a lot of information, it doesn't waste anything it actually uses.

Answer 3

For the Python implementation of the two-sample chi-squared test:

from scipy.stats import chi2_contingency
import numpy as np
arr1 = [359, 387, 187, 48, 12, 7]
arr2 = [599, 311, 75, 14, 1, 0]
obs = np.array([arr1, arr2])
chi2, p, dof, ex = chi2_contingency(obs)
#Returns: chi-squared value, p-value, degrees of freedom, expected counts