Permulations to test if values are different than zero?

Question

I have a distance measure, which ranges between 0 and 1, between several pairs of populations. I need to figure out if each distance is statistically different from zero using permutations, any suggestions on how to achieve that? Thanks

Answer 1

You may need to expand your description.

We generally do not use the phrase "statistically different" when referring to individual values, if a number is not 0, then it is not 0, statistically or otherwise.

When we use "statistically different" it is in reference to an unobserved parameter, such as the mean of a distribution/population. So if you have a sample of differences and want to know that the mean difference is 0 or not 0 then we can talk about statistical significance. In your case, if all data is restricted to be between 0 and 1 then if any observed value is >0 then the p-value for testing the overall mean = 0 is 0 (very significant).

If your goal is to compare if your 2 groups have the same mean (median, or other parameter) then you can use a permutation test. First find the difference in the 2 sample means (medians, etc.), now pool all the observations together and randomly divide them back into the 2 groups (of the same sizes as the original) and find the difference in these random groups. Repeat the process a bunch of times (e.g. 9998 times). Now look how the difference in the real data compares to the distribution of differences from all the sets. The p-value is the proportion of differences that are as, or more extreme than the original.