0

I am unsure as to how to compute a two sided p value following permutation testing, following different examples online.

For example, this post Two-sided permutation test vs. two one-sided, gets the proportion of the absolute correlation coefficients from the permutation test greater than or equal to the absolute non-permuted correlation coefficient.

When I ran this, here were my results;

     set.seed(1)
     x <- runif(20)
     y <- 0.5 * x
     y <- y + rnorm(20)
 #set up for the permutation, compute observed R
 nullR &lt;- numeric(length = 1000)
 nullR[1] &lt;- cor(x, y) ## obsered R in [1]
 N &lt;- length(x)

#permutation test
for(i in seq_len(999) + 1) {
     nullR[i] &lt;- cor(x[sample(N)], y)
 }

>one side H1 R > 0 >sum(nullR >= nullR[1]) / length(nullR) [1] 0.929 >one side H1 R < 0 > sum(nullR <= nullR[1]) / length(nullR) [1] 0.072 > two sided > sum(abs(nullR) >= abs(nullR[1])) / length(nullR) [1] 0.155

Whilst, this website (https://dgarcia-eu.github.io/SocialDataScience/5_SocialNetworkPhenomena/056_PermutationTests/PermutationTests) calculates a two sided p value following permutations of correlations as:

         p_value_Cor <- (sum(nullR>=nullR[1])+1)/length(nullR)

which gets the same value as the first one sided test 0.93.

Q1) which is the right way to calculate a two sided p value following permutation of correlations?

Q2) if I used cor.test, which would get the test statistic and estimate, which should I use to calculate the p value and why, the estimate like they have done above or the test statistic? In my own data, I would be using Spearman correlation.

Any clarity would be appreciated.

aim6789
  • 55
  • The + 1 is common. It ensures that your p-value will never be an exact 0, regardless of how many (or, more importantly, few) permutations you perform. – Roland Mar 15 '23 at 06:22
  • @Roland thank you for your comment - was wondering whether you would choose the test statistic or the estimate (like in the example above) to calculate the p value? – aim6789 Mar 15 '23 at 16:44

0 Answers0