3

I have approximately 5,000 variables, each with 3 data points. These numbers are ratios, i.e. bounded at 0 and generally non-normal. I'm interested in testing whether the mean of each of the 5,000 variables is significantly different from 1.

Is there a test that's suitable here, given that the sample size is tiny (n=3) and the data isn't normal? Would a Mann-Whitney U-test work?

More generally, does it even make sense to assign a p-value to such tiny sample sizes?

Edit: In case anyone's interested, my solution was to log-transform the data so it's (at least approximately) normal, and then use a t-test. A parametric approach seems like a good way to maximize what little power I have. There are likely many others, but the discussion here was helpful.

R Greg Stacey
  • 2,302
  • Maybe your problem is not so bad at all: "Sample size" refers to the number of independent subjects, the independent replications. If each measurement (maybe consisting of three numbers) is independent from the others, you have a large sample size. – Horst Grünbusch Feb 26 '16 at 23:57
  • Sorry, I worded that badly. I actually have 3 independent measurements (3 separate samples, different times) and each measurement consists of 5,000 numbers... So I want to perform 5,000 tests, each on 3 data points. – R Greg Stacey Feb 27 '16 at 00:04
  • I changed the wording in the question to reflect this. Thanks – R Greg Stacey Feb 27 '16 at 00:16
  • The Mann-Whitney U-test is asymptotic. I would not suggest to use that. For very small nonparametric samples, permutation tests are best suited. I doubt that they would show any significance, though. Maybe you should replace the mann-whytney-u-test tag by the permutation-test tag to attract more attention. – Horst Grünbusch Feb 27 '16 at 00:24
  • Thanks! I don't see a 'permutation-test' tag. I added the 'permutation' tag, but I'm not sure if that's appropriate (and it has 3 followers...). – R Greg Stacey Feb 27 '16 at 00:29
  • @Horst the actual Mann-Whitney U test is not asymptotic -- it's exact small sample (read their paper, where they derive the small-sample distribution under the null). The real problems with it are (aside from the main issue with permutation tests in my answer) -- 1. it's not a test for means. 2. it's two sample but the problem is one-sample – Glen_b Feb 29 '16 at 23:28
  • @Glen_b: I learnt to call the exact test "Wilcoxon" and the asymptotic procedure "Mann-Whitney" which makes some sense, because the original work from Mann and Whitney concludes approximative normality for sample sizes $>8$. However, following your note I found a paper by Kruskal saying that the procedure was first discovered in 1914 by Gustav Deuchler. Interesting! – Horst Grünbusch Mar 01 '16 at 15:54
  • @Horst While I can understand the reasoning, that usage is not widespread -- if you use "Mann-Whitney" to refer only to the asymptotic normal approximation, your intent is very likely to be misunderstood. (Yes, there were indeed some related papers before Wilcoxon's paper, and several more about the 1945-47 time period between that one and the Mann & Whitney paper. The origins of the test are not simple) – Glen_b Mar 01 '16 at 21:49

1 Answers1

2

The problem with any permutation-based test* is that with three observations you only have 6 possible arrangements. With a two-tailed test your smallest possible significance level is $\frac{1}{3}$ !!

Mostly people prefer their type I error rates not to be quite so high.

Low power (high type II error) will also be a problem.

A parametric approach, if one can reasonably be supported, is pretty much** your only hope at n=3. You will be quite reliant on its assumptions.

* note that the Mann-Whitney (and one-sample rank tests that are more nearly relevant here) is also an example of a permutation test, one where the statistic is based on ranks.

** however, perhaps surprisingly, it is possible to do something even at n=1, though the power isn't great

Glen_b
  • 282,281
  • Thank you! That's the conclusion I arrived at after a bit of reading (and naively wondering why I couldn't get a significant p-value with the signed rank test!). – R Greg Stacey Feb 29 '16 at 23:31
  • And with a sample size of zero you just need a prior. :) – dsaxton Feb 29 '16 at 23:44
  • @Qroid the signed rank test also assumes symmetry under the null. When you have it, the power relative to the one-sample t-test - even at the normal, even in small samples - can be quite good, as long as you can tolerate the high significance level. – Glen_b Feb 29 '16 at 23:44