2

I have two related samples, for which I want to prove they are not significantly different (normally you would test for the opposite, i.e. samples are significantly different).

If I use Wilcoxon signed-rank test, the test results are of course highly insignificant, with p-values greatly above 0.5, sometimes even above 0.9.

So in an ideal world, I would use some sort of inverted Wilcoxon signed-rank test (with negated null hypothesis). But to my knowledge, this kind of test does not exist.

I have also been looking for a way to reject the alternative hypothesis OR accept the null hypothesis (for the Wilcoxon signed-rank test), but also found that this is a no-go.

Are there any other options regarding this?

Some background: I have two different methods from two different authors, and I want to state that since there are no differences between them, one should use the simpler one.

Update: below you can find some sample data for both methods - method 1 in column 1 and method 2 in column 2. Values are dependent (paired). Columns separated with a tab.

0.045069233 0.044114038
0.04769785  0.047292581
0.051955983 0.052377575
0.047711922 0.048879883
0.044005404 0.044643139
0.045603963 0.047935382
0.048257908 0.048353034
0.045589891 0.045802094
0.043481932 0.043902961
0.049853653 0.05026849
0.044545762 0.045065293
0.041357087 0.041142069
0.04399696  0.043582123
0.046656535 0.048246651
0.046656535 0.0463402
0.045066419 0.044957785
0.042950017 0.043476303
0.049302038 0.047615108
0.05143814  0.052491838
0.036578296 0.036366093
0.047723179 0.047617922
0.047725993 0.046558032
0.045074862 0.045594394
0.043473489 0.044952719
0.046648092 0.045068108
0.047714736 0.047401779
0.043473489 0.043470111
0.0546043   0.053232016
0.042947203 0.043152651
0.042429359 0.043155465
0.047191264 0.048875943
0.040813914 0.041778116
0.049302038 0.0488872
0.046121806 0.045916357
0.04559552  0.045063605
0.050889339 0.05005291
0.045083305 0.044008781
0.036049195 0.035844872
0.050365867 0.049311606
0.044002589 0.044532815
0.04718845  0.046546775
0.048775751 0.045383316
0.047714736 0.048045705
0.040284814 0.038278172
0.043999775 0.047074187
0.044534504 0.04474333
0.046127434 0.046334572
0.030217832 0.029900934
0.042412473 0.043476303
0.04240403  0.043900709
0.042423731 0.041353146
0.049310481 0.048038388
0.039764156 0.039553642
0.051440955 0.050577508
0.04984521  0.048767871
0.048257908 0.04729427
0.042398401 0.043157717
0.041348643 0.040503771
0.047711922 0.047933694
0.044540133 0.043583249
0.048778566 0.049092086
0.041883373 0.042201396
0.045052347 0.045284251
0.046664978 0.045174491
0.047731622 0.048883823
0.045589891 0.045492514
0.046116177 0.045810537
0.051964426 0.051527074
0.043484746 0.045594394
0.044545762 0.045281436
0.032337048 0.029580097
0.04718845  0.047612856
0.047199707 0.046657098
0.047182821 0.047504222
0.037104582 0.036789936
0.032339863 0.032554317
0.036575481 0.036157267
0.038176855 0.037964089
0.05141844  0.050258359
0.045055162 0.045488574
0.046645277 0.047825622
0.048767308 0.049621749
0.052487898 0.051109422
0.035536981 0.034679725
0.048241022 0.04729427
0.047191264 0.046766295
0.051429697 0.050153664
0.047202522 0.046343578
0.039229427 0.039237307
0.045587076 0.044001463
0.049316109 0.050368119
0.049831138 0.051958798
0.043985703 0.045915794
0.046645277 0.04592086
0.038691883 0.039340876
0.044016661 0.045705843
0.039766971 0.039765845
0.048232579 0.046756163
0.038722841 0.037751886
0.04876168  0.047928065
alesc
  • 416
  • 1
    For samples, you don't have to "prove" anything. For populations, you cannot "prove" anything, you can do just reasonable decision. – ttnphns Feb 26 '15 at 15:27
  • Poor choice of words on my side, but this does not change the problem I am trying to solve. – alesc Feb 26 '15 at 15:34
  • Are you aware that the word "significant" cannot appear in hypotheses, at least not in its usual meaning? – Michael M Feb 28 '15 at 12:28
  • This won't be in any hypothesis. It is just to prove a point that the simpler method is as good as the more complex and supposedly better better (by claims of it's author). I have done a lot of data crunching for this part and there really is no difference between the both methods. Now I am looking a way to formally write this down, hence my post and bounty. – alesc Feb 28 '15 at 13:09
  • 1
    You're trying to prove the hypothesis (distibutions are not different) which cannot be given strict formulation and thus cannot be proved.

    As an example: one distribution is gaussian distribution, the other one is obtained by drawing a number from gaussian distribution and truncating it to 10 hexadecimal symbols. These distributions are different? From one point, yes - you almost surely can distinguish samples from these distributions. From the other point, their CDFs are almost equal (and all tests will fail to distinguish them).

     – Alleo Feb 28 '15 at 17:22
  • I agree with @Alleo above but want to add that I see no issue in just reporting your p-value and saying that you failed to reject the null. Is a difference between 4 and 4.5 a big or small difference in your domain? What about 4 and 4.00001? I think searching for a specific (and therefore maybe somewhat obscure) statistical test will just obfuscate your point. – bill_e Feb 28 '15 at 19:36
  • 2
    I'd compute absolute values of the differences and analyze them descriptively, maybe adding an upper confidence limit for the true average difference. – Michael M Feb 28 '15 at 21:16
  • There are detectable differences between the two columns. For instance, when column 1 is split according to whether column 2 is greater or less than it, the two halves of the split have significantly different variances ($p=0.0016$). This was found by studying only the 50 even-indexed observations and confirmed on the other 50 observations. The point is that "not significantly different" is never true; you need to stipulate exactly what kind of quantitative difference you wish to test and to specify how large a difference would be of no importance to you. – whuber Mar 01 '15 at 16:59
  • Question to the background: What means "no difference"? See my comments to the answers below. The procedures in these answers are quite different, so I suggest you should be more specific which kind of difference you focus on. – Horst Grünbusch Mar 02 '15 at 17:29

2 Answers2

5

The "inverted test" as you call it is commonly known as equivalence test and a weaker version of it is called noninferiority test. I dont know the answer to your question: you want a nonparametric paired equivalence test.

This question in CV non-parametric two-sample equivalence tests with unequal sample sizes deals with non paired tests. If you can accept the weaker, non-paired test, maybe you can start from that. If you can drop both the paired and the non-parametric requirements, a known equivalent test is TOST (two one-sided t-tests). It is implemented in R in the package equivalence (see the function tost http://www.inside-r.org/packages/cran/equivalence/docs/tost The tag in CV may point you to other answers.

Equivalence tests have a further complication, that the null hypothesis is not that the difference is 0 but that the difference is smaller than a threshold of irrelevance - that is, the difference is below a level that is irrelevant for practical purposes. You will have to define this level of irrelevant difference based on your knowledge of the problem. This is the parameter epsilon in the function tost I pointed above.

Community
  • 1
Jacques Wainer
  • 5,387
  • The raw data from my question post are subsequent runs of both methods, where each line has the same input for both methods. That's why I treat them as paired data. I could probably reshuffle the values and treat them as non paired. I could even drop the nonparametric requirement, if all assumptions are met. I will look into your suggestions tomorrow. Thank you for pointing me in another useful direction. – alesc Mar 01 '15 at 08:34
  • There is no need to reshuffle. If you use tost it will treat your data as 2 sets of data - with no pairing. I also realized that you can compute the difference of each line in the two datasets and use a one-sample equivalence test on the difference. I dont know anything about one-sample equivalence test, not eve if it exists, but if the post above explicitly talks about two-sample, so it is likely that there should be a one-sample. – Jacques Wainer Mar 01 '15 at 20:57
  • This approach would test if the distributions are similar, or more precisely, that $0.5-\epsilon < P(X_1>X_2) < 0.5+\epsilon$, if $X_1$ is a value of the first method and $X_2$ is the second method's value on the same subject. $\epsilon$ denotes some equivalence margin. So it tests that one method does not systematically yield higher values than the other method. – Horst Grünbusch Mar 02 '15 at 17:19
  • I have looked a bit more into this mater and found two viable options: the TOST and the ICI (Inferential Confidence Intervals). The problem I am facing with TOST is that I have to compute the difference between the variables, which, strangely isn't normally distributed (and each of the variable is). Regarding my problem, is it best to use ICI? – alesc Mar 05 '15 at 09:44
1

Can you treat this as an Interrater reliability test? The logic is if the two methods are equally well, then they should reliability produce similar results, then you can use Intra-class correlation coefficient (or Kappa if you have nominal data), or other interrater reliability test.

ceoec
  • 534
  • 1
    Yes, actually if both methods produce similar results, then they should also agree. But will this be applicable to decimal numbers? As far as I've glanced over this topic, the Cohen's Kappa is only for categorical data. It also works with ordinal data, but I don't know if it makes sense because it is highly unlikely that two decimal numbers will be the same. I have also updated my question with some sample data. – alesc Feb 28 '15 at 18:53
  • 1
    Looks like you can try Intra-class correlation coefficient -- I suggested Kappa as you tagged nonparamatric... – ceoec Feb 28 '15 at 20:28
  • I currently treat values as nonparametric, because I am not sure if the assumptions of let's say Student's T-test are met. In other words, I rather play it safe than violate any assumptions. @ceoec: would Kappa be applicable for my values? – alesc Mar 01 '15 at 08:26
  • I think ICC would be a better choice for your data. Kappa is for nominal data as you also pointed out... – ceoec Mar 01 '15 at 17:09
  • This approach would state reliability even if the values of one method are systematically higher than the values of the other method. Also, in order to infer similarity, you would have to set up some kind of equivalence bound. – Horst Grünbusch Mar 02 '15 at 17:27