I am unsure as to how to compute a two sided p value following permutation testing, following different examples online.
For example, this post Two-sided permutation test vs. two one-sided, gets the proportion of the absolute correlation coefficients from the permutation test greater than or equal to the absolute non-permuted correlation coefficient.
When I ran this, here were my results;
set.seed(1)
x <- runif(20)
y <- 0.5 * x
y <- y + rnorm(20)
#set up for the permutation, compute observed R
nullR <- numeric(length = 1000)
nullR[1] <- cor(x, y) ## obsered R in [1]
N <- length(x)
#permutation test
for(i in seq_len(999) + 1) {
nullR[i] <- cor(x[sample(N)], y)
}
>one side H1 R > 0
>sum(nullR >= nullR[1]) / length(nullR)
[1] 0.929
>one side H1 R < 0
> sum(nullR <= nullR[1]) / length(nullR)
[1] 0.072
> two sided
> sum(abs(nullR) >= abs(nullR[1])) / length(nullR)
[1] 0.155
Whilst, this website (https://dgarcia-eu.github.io/SocialDataScience/5_SocialNetworkPhenomena/056_PermutationTests/PermutationTests) calculates a two sided p value following permutations of correlations as:
p_value_Cor <- (sum(nullR>=nullR[1])+1)/length(nullR)
which gets the same value as the first one sided test 0.93.
Q1) which is the right way to calculate a two sided p value following permutation of correlations?
Q2) if I used cor.test, which would get the test statistic and estimate, which should I use to calculate the p value and why, the estimate like they have done above or the test statistic? In my own data, I would be using Spearman correlation.
Any clarity would be appreciated.
+ 1is common. It ensures that your p-value will never be an exact 0, regardless of how many (or, more importantly, few) permutations you perform. – Roland Mar 15 '23 at 06:22