Multiple Test Correction: Can we divide dataset into subgroups?

Question

Assume that the researcher wants to test huge amount of hypothesis and present his discoveries. He has data from two groups, eg, 100 people vs 100 people, and he wants to

1) having 20.000 genes expression measured from each person, figure out how many of genes are differentially expressed

2) having 1.000 different metabolites, figure out how many of them show different levels

3) having 100 physiological characteristics, such as height/weight/body temperature/etc, figure out which are different between groups

So in the end a person performs 21.100 tests. My question is: what is a correct way to deal with the resulting p-values - should I apply FDR correction on each set of tests 1)-3) separately, or all together? If I apply separately, I will have much bigger power in case 3) than in case 1). If I apply jointly, I will have a really low power for any case. If separate FDR control is possible - why can not we make it absurd and divide our 21.100 tests into groups of size 10 or even 1? What if I have different cohorts of 100 vs 100 people for each situation 1),2),3) - but I want to present this results in one paper? The answer seems to be clear when the groups do not have positively dependent p-values - so mixing of them may ruin typical FDR control assumptions - but what if p-values are eg independent?

I feel that it has something with Simpson's paradox, but can not understand it...

UPD: I found this question already asked, even several times, on stats.stackexchange - (so not only me who cares about that and there is a huge off-line confusion with this analysis among my colleagues who are not statisticians at all), but there is no answer

UPD1: The obvious answer may be formulated as "it depends on which proportion of false discoveries you want to guarantee - inside each group or overall". I guess when researchers publish a paper, they are mainly interested in the overall percentage of false claims throughout the paper.

UPD2: Could you as statisticians make kind of guideline on reporting results in this sense, which accents should be pointed while reporting bunch of statistically significant results devoted to different phenomena within one paper? My knowledge is enough to give only the answer from UPD1...

UPD3: if you downvote the question, please, elaborate in comments why you think it deserves to be closed and why there is nothing to discuss. I would be glad to know that the answer is obvious, point me (and a lot of people who are also confused) at this answer.

Answer 1

I'm going to offer a couple of strategies to consider, and they are purposefully "simplistic" in nature, as there is no "statistical" answer that will provide the pragmatic answers you probably are hoping for. Additionally, I, as an applied statistician, would recommend looking at tests 1 thru 3 separately, as they appear to be different analyses (even with some amount of shared data). This is conventional in many disciplines, and I don't think you would encounter much push back (as the pragmatic counter-argument is the alpha as error rate vs. family-wise error rate vs. life-time-wise error rate).

Approach #1 (MCP/FDR):  Holms-Bonferroni
This is the approach that would probably "clear the reviewers". It will mean a lot more low p-valued tests being flagged as n.s. (as you suggested above).

Approach #2:  Pick an absolute cut-off
Using some metric of effect size, say that you will consider any tests that cross this threshold as "reportable" and you will report an unadjusted $p$-value for interpretation purposes of those researchers who wish to use your findings in support of their future studies. If you go this route, the cut-off should be a fairly high hurdle to cross.

Approach #3:  Just pick a strict $\alpha$ and stick with it
Here's the logic for the strict alpha approach: If I'm going to conduct 20,000 tests, I'm not expecting them all to be correct. So, ¿how many are ok to be wrong? The familywise error approach says zero (despite the fact that the very nature of the evidence we are utilizing has a built in uncertainty to it). So, let's say we are ok with 1 error out of 20000 (or in probability terms at most 1 error). Then we could use an $\alpha = 0.000\ 002\ 5$. This is the probability that produces $P(X \le 1) > 95\%$. With this logic, if we are willing to accept at most 2 errors in 20,000 (a 0.01% error rate), $\alpha = 0.000\ 040\ 8$. For at most 5 errors, $\alpha = 0.000\ 130\ 6$. (For at most 20 errors, $\alpha ≈ .0007$.) The rationale is simple: there will be some errors, but there are at most $k$ of them...and it is the job of future researchers to determine which ones, of our the 20,000 tests, are wrong.

Approach #4:  Just report the results
The very idea that statistical/probabilistic assessment provides a dichotomization of evidence into truth-affirming or truth-denying fact is inherently flawed. No one would expect 20,000 different journal articles to contain no errors. But, the users of said 20,000 journal articles operate as such, regardless of the fact that a consistent $\alpha=0.05$ would mean ≈1,000 are actually wrong.
       This goes against some of the basic tenants of the scientific paradigm. We cannot prove, only disprove. Thus, if we have evidence that is suggestive, (say, god-forbid, a $p=0.051$), then it is worthy of further exploration. If you reported the results without any adjustment, and included the caveat that this information is provided to suggests possible avenues for future research, then the 20,000 tests would add more evidence to our growing knowledge base. True, it will be flawed evidence...but in the end, ¿what evidence isn't flawed in some manner?

Not sure if this helps or not, but I'm sure it will spur on some amount of discussion. As always, happy to clarify anything when possible.