2

I'm dealing with a statistical problem, where I tested a bunch of hypotheses with a very strong positive dependence within certain groups of them. Many of them didn't lead to significant results. Now, the pure number of those positve-dependent insignificant results leads to all other null-hypotheses being accepted as well by the Benjamini-Hochberg-procedure, even though they clearly look wrong when looking at the actual data, so I believe that acceptance of those is overly conservative.

Is there anything I can do about this to still extract some significant findings here?

mizanshu
  • 23
  • Could you please explain what you mean by "positive-dependent insignificant hypotheses"? Do you perhaps mean that the data used for testing two or more hypotheses are not independent? If so, then you need to provide the specifics: how, exactly, are these data (and the tests based on them) dependent? – whuber Nov 21 '23 at 18:39
  • 1
    Are you sure you use the terminology correctly? Normally null hypotheses are rejected or not, the term "rejected" does not apply to the alternative. But you say "I believe that the rejection of those is overly conservative" - the term "conservative" means that rejection is harder and fewer null hypothesis are expected to be rejected by a conservative test. – Christian Hennig Nov 21 '23 at 18:42
  • "even though they clearly look true when looking at the actual data" - how exactly can you see this? In fact no model or hypothesis is ever literally "true". Of course data may not look like any indication against the null hypothesis, but even this can be deceptive, particularly with large sample sizes where small deviations from the null hypothesis can easily lead to rejection. (As said above I'm not actually sure whether it's the null hypothesis you're talking about here.) – Christian Hennig Nov 21 '23 at 18:46
  • The alternative will never be rejected. – Christian Hennig Nov 21 '23 at 18:46
  • Note also that you'd need to try any alternative to what you have already done on new independent data, as choosing another test for the reason that an earlier test didn't give you the significances you hoped for is invalid (the theory behind these tests doesn't allow you to pick a test dependent on other results on the same data). – Christian Hennig Nov 21 '23 at 18:49
  • The essence of multiple testing is that if you look at many things, some look "significant" by accident even if this doesn't mean anything. This doesn't only apply to tests, also it applies to looking at the data. So without any further information, (e.g., visualisation) I'm not convinced that the data really show that the null should be rejected and non-rejection is due to conservativity. It may well be that your visual diagnosis is affected by the multiple testing problem. – Christian Hennig Nov 21 '23 at 18:57
  • Any relevant information should be in the question, not in the comments. – Christian Hennig Nov 21 '23 at 18:58
  • 1
    This is not personal. What I want to believe or not is irrelevant. You don't present any evidence. You yourself write that you're "new to this", so on what basis could I take for granted that your "clearly look wrong" assessment is correct? The points that I am making are important to understand when doing analyses like this. I stop commenting now anyway. – Christian Hennig Nov 21 '23 at 19:08
  • The meaning of independence in any application of multiple corrections testing, like B-H, is that the data are independent between one test and the next. It says nothing about the likelihoods related to the parameter. That suggests you might be conceiving of this in a Bayesian way where you have some idea of a prior distribution of the parameter -- but so far, nothing in your post says anything about this crucial assumption. Moreover, correcting for multiple hypothesis tests is irrelevant in a Bayesian setting, leaving us wondering what you are assuming and what's really going on. – whuber Nov 21 '23 at 20:07
  • It seems there is a lot of commentary here which is missing from the original question. Please edit your question to include these additional details, as many answerers may skip this section (and often do) because they get lost in a sea of comments. Editing this into the question provides a lot more context anyway. – Shawn Hemelstrand Nov 21 '23 at 22:42

1 Answers1

5

Before getting tied up in questions of how to deal with the power-robbing effects of 'corrections' for multiplicity, you should first think about whether your inferences will be improved by such adjustments. This topic has been addressed many times on this site. Here are a couple of recent examples, and a quick search will reveal many more.

p-value correction after wilcoxon/mann-whitney test

How many p-value observations do you think are required before doing FDR correction

Michael Lew
  • 15,102