8

Out of curiosity, I realised that the Kolmogorov-Smirnov normality test returns two very different p-values depending on whether the dataset has small or large numbers. Is this normal and is there a number size limit for this test? From what I saw, the Shapiro-Wilk test was much more stable.

I tried this

ks.test(c(0.5379796,1.1230795,-0.4047321,-0.8150001,0.9706860),"pnorm")

One-sample Kolmogorov-Smirnov test


data:  c(0.5379796, 1.1230795, -0.4047321, -0.8150001, 0.970686)


D = 0.3047, p-value = 0.6454


alternative hypothesis: two-sided

And then I multiplied each value by 100



ks.test(c(53.79796,112.30795,-40.47321,-81.50001,97.06860),"pnorm")


One-sample Kolmogorov-Smirnov test




data:  c(53.79796, 112.30795, -40.47321, -81.50001, 97.06860)


D = 0.6, p-value = 0.03008


alternative hypothesis: two-sided

With the same data, the Shapiro-Wilk test returns a p-value of 0.3999.

Nick Cox
  • 56,404
  • 8
  • 127
  • 185
Xav64
  • 169

2 Answers2

17

The ways you’ve coded it, you’re asking the KS test about a null hypothesis that the distribution is $N(0,1)$. In the first set of numbers, that looks plausible. Consequently, the p-value is high. In the second set of numbers, that does not seem to be the case. Numbers like those don’t typically come from a $N(0,1)$ distribution. Consequently, the p-value is low.

By multiplying by a factor, you’ve changed the variance. Since the KS test considers all aspects of the distribution, variance included, the test correctly regards the two data sets as different.

The reason that Shapiro-Wilk is more stable is because it evaluates normality. Multiplying by a positive factor does not change the normality, so Shapiro-Wilk will not have the same kind of sensitivity to a variance change that KS has.

Dave
  • 62,186
  • 1
    True, there was a typo in the code for the *100. but your answer still holds, thank you ! – Xav64 Mar 06 '23 at 09:44
  • 3
    In addition, there are specific warnings about using the K-S test with parameters estimated from the data, which you presumably need here, and general warnings about the relative uselessness of any such tests. Just plot the data on a normal quantile plot! (Many other names: normal probability plot, normal scores plot, etc.) – Nick Cox Mar 06 '23 at 09:54
  • Thanks. If we still want to run this test, do we have to systematically standardize our data first? – Xav64 Mar 06 '23 at 09:57
  • 2
    See this for elaboration on what @NickCox wrote about normality testing being of questionable usefulness. // Regarding issues with testing with estimated parameters, check out the Lilliefors test. – Dave Mar 06 '23 at 09:57
  • 1
    @Xav64 Standardizing your data with the sample mean and variance and then testing if the standardized data are $N(0,1)$ is the same as testing with the original data and using the sample mean and variance. You can check this in a simulation where you will get identical p-values. (I assume you mean the usual z-score by standardizing: $x_{new}=\dfrac{x_{original}-\bar x}{s}$.) – Dave Mar 06 '23 at 09:58
  • @Dave thanks. Do you know how to parameter the kolmogorov test to directly test with the original data and using the sample mean and variance? – Xav64 Mar 06 '23 at 10:06
  • 2
    I don’t totally follow what you’re asking, but that Lilliefors test might be what you want. @Xav64 – Dave Mar 06 '23 at 10:09
  • 1
    Yet more: the K-S test is necessarily better at comparing middles of distributions than at comparing tails. In practice, if deviations from normality are important, I would say it is usually because of what is happening in the tails. – Nick Cox Mar 06 '23 at 10:19
8

Adding to the existing response, it's worth noting that the two ks.test calls below produce the same output.

x = c(0.5379796,1.1230795,-0.4047321,-0.8150001,0.9706860)
ks.test(x, pnorm)
#> 
#>  Exact one-sample Kolmogorov-Smirnov test
#> 
#> data:  x
#> D = 0.3047, p-value = 0.6454
#> alternative hypothesis: two-sided

ks.test(x100, pnorm, sd = 100)
#> 
#>  Exact one-sample Kolmogorov-Smirnov test
#> 
#> data:  x  100
#> D = 0.3047, p-value = 0.6454
#> alternative hypothesis: two-sided

R syntax note: the default arguments to pnorm() are mean = 0, sd = 1. Anything after the second argument in ks.test() gets passed as an argument to the pnorm() function in this case.

Eoin
  • 8,997
  • so does it mean that the optimal way of running this test is to write ks.test(x, pnorm, mean=mean(sample), sd=sd(sample)) or is there a faster way? – Xav64 Mar 06 '23 at 12:04
  • 6
    Reply to comment: 1. No because that test is invalid. See remarks elsewhere with key word=:Lilliefors. 2. Who said optimal=faster? – Nick Cox Mar 06 '23 at 12:40