3

Say I have two devices that purifies milk, device A and device B. I assume Device B is better and it produces milk that is 5% better than A. If I want to test this assumption, I take 8 sample from the milk produced by Device A and 8 samples from Device B, analyse the purity of the samples taken at the lab, and compare the results of 8 sample from A and the 8 samples from B. The question is: What is the best statistical test to use? Do I need to run power analysis? How to check normally when we have only 8 samples?

This question and this are similar but I could not make use of the answer there.

owise
  • 165

1 Answers1

2

Well, if I understand your problem correctly, you want to test if the purity of samples produced by device B is smaller than the ones from device A.

I would use Mann–Whitney U test .

Null hypothesis (H0): the values of A and B come from the same distribution.

The test show you if one of the samples tend to have values larger than the other.

EDIT (thanks to whuber for point out the 5% test): Formally, you could try to test if you can disprove that device B is 5% better than device A (you would do that by taking H0: B better than A 5%). Now, this strategy has two problems. One is that, if normality doesn't hold, it is really hard to test it. By the way, you can check normality with Shapiro–Wilk test. The second problem is that, if you can't reject the null (meaning that there is no support for a difference different than 5% you still have not proven that is 5%).

My humble solution would involve bootstrap and confidence intervals. You can use bootstrap to compute a distribution of the ratio of the means (or medians). You can do this in R very easily. Them you have a proper confidence interval for how better B is from A.

Diogo Santos
  • 787
  • 3
    How do you propose adapting this to the hypothesis that was stated; namely, that one is at least 5% better? – whuber Sep 21 '18 at 21:56
  • Ok, I did not understood that. I thought that the 5% extra here just an example. I will think about it and edit my answer. – Diogo Santos Sep 22 '18 at 08:30
  • Can Shapiro–Wilk test be used to test normality in such small sample size cases? Can you please point out to the test name? is permutation test? –  owise Sep 22 '18 at 10:38
  • You can use Shapiro-Wilk test, but it's power is small (as for all tests, see [this paper]https://www.nrc.gov/docs/ML1714/ML17143A100.pdf . Which test do you refer? I'm pointing you to Mann-Whitney U test. Remember that, given it's weak assumptions, if you manage to reject the null them for sure one is producing better milk than the other (this is not an answer for the 5% specific problem). – Diogo Santos Sep 22 '18 at 21:34
  • 16 values is too small to justify the bootstrap. When sample sizes are small, one has to make relatively strong assumptions in order to draw conclusions. That usually means parametric assumptions. In this case, if the purity has to be checked repeatedly, a database of previous measurements can establish and support distributional assumptions, obviating any need for normality testing. – whuber Sep 23 '18 at 15:49