0

I am running an experiment where I get 22,152,480 p-values in a given order, ranging from 0 to 1, but strongly tilted toward 1 (i.e. not uniformly distributed). I want to say that a given series of them is significant between two samples.

For example, the p-values for given tests are 0.0428, 0.0585, 0.0308, 0.0342, 0.124. It is like rolling dice, the probability of rolling any particular number over and over again is extremely low with the more rolls there are, about (1/6)^n, where n is the number of tests. Similarly, using a p-value cutoff of 0.05 means that the p-value of finding that many tests in a row decreases even more quickly, (0.05)^n, here the p-value product is 3.27e-07

I was told that the best way would be to combine all possible products of n pvalues of the number of p-values, i.e. 22,152,480^n which is impossible to do on a computer.

Would Fisher's method be appropriate here?

con
  • 133
  • Con, it seems to me quite likely that the most useful answer to your question is an explanation of why you shouldn't, or can't, or shouldn't want to do what you're asking for. Can you give more details? What are the data? How were the samples obtained? What are the 22 million tests that gave you the P-values? What is your scientific objective? – Michael Lew Nov 09 '17 at 19:57
  • @MichaelLew the data are genomic CpG obtained by sequencing. Each is presented as percent methylated or percent unmethylated. The scientific objective is to identify regions which are differentially methylated. – con Nov 09 '17 at 21:15
  • Are the P-values from significance tests? What do the 22 million samples tested consist of? – Michael Lew Nov 09 '17 at 22:32
  • the 22 million samples are individual CpG nucleotides in DNA. These are significance tests, calculated like t.test(d1,d2, alternative="two.sided", var.equal=FALSE) in R – con Nov 10 '17 at 15:58
  • If your objective is to determine which areas of the DNA are "differentially methylated" and you are measuring the methylation then it is not clear what role you want P-values to play. It is also not clear what you are trying to say in your second paragraph. What are the 5 P-values mentioned in the second paragraph? It is also not clear what are the significance tests that you say you are using. What are "d1" and "d2"? What are the sample sizes for d1 and d2? Why would you want to combine 22 million P-values? If you will not provide details of your design then you will not get useful advice. – Michael Lew Nov 10 '17 at 20:36

1 Answers1

1

With such a large number of $p$-values almost any method is going to be likely to reject the null hypothesis that they are drawn from a uniform distribution. There are a number of methods which you can use and some simulations by Loughlin in a paper here entitled "A systematic comparison of methods for combining p-values from independent tests." may be of interest. One feature not brought out in his paper is that responsivenss to deviations in both directions is handled differently by the methods so if your $p$-values were (0.001, 0.001, 0.999, 0.999) some methods (including Fisher's) will give a significant value while others (including Stouffer's and Edgington's) return 0.5.

mdewey
  • 17,806
  • is there a way I could get a distribution function for all numbers in a set (0,1] in a set multiplied by each other n times? I could integrate this to get an adjusted p-value. Something like y = x^2 or similar. That is, when there are products of n numbers, all numbers up to x sum to some % of the total – con Nov 10 '17 at 16:06
  • Fisher's method is the product of the $p$ -values. Why do you want to raise them to some arbitrary power first? – mdewey Nov 11 '17 at 13:45