Are methods of p-value adjustment only necessary when a universal/overall hypothesis is the target of inference?

Question

I am trying to understand more clearly in which cases an adjustment of p-values is necessary. At the moment my reasoning about this can be summarized as follows:

Benjamini and Hochberg (1995) list three problems/scenarios where adjustment of the type I error is called for:

1. multiple outcomes ("multiple end points problem"),
2. multiple comparisons ("multiple-subgroups problem") and
3. "screening problem" (e.g. screenin of multiple predictors).

Some, for instance Perneger (1998) have argued that such adjustment is necessary only in cases in which one wants make an inference about a "universal hypothesis" from multiple tests.

To me, all three of the above scenarios can be understood as such a case: In scenario 1) the overall (=univeral) hypothesis is that the treatment has an effect (on at least one of the outcomes). In scenario 2) the overall hypothesis is that there are (any) differences between conditions. In scenario 3) the overall hypothesis might be that any of the predictors affects the outcome (although that logic is less clear to me in this case).

So does that mean that drawing an inference about such an overall hypothesis (using several seperate tests) is the main/real/only reason for using p-value adjustment methods?

And does that mean that if I strictly avoid inferences about an overall hypotheses of any kind (directly/implicitly or otherwise), then there is no reason to use such adjustment methods?

Or am I misunderstanding something here (and if so, what)?

Answer 1

Think of it like this. In a one-way ANOVA setup, you are testing whether the population means of more than two independent groups are equal. Clearly, if there are only two independent groups, the testing problem will simply be an independent sample t-test. Otherwise, if you found significance in ANOVA, you have to proceed with a pairwise t-test with p-value adjustment methods. That means if you don't have to test a universal hypothesis using multiple testing methods, the p-value adjustment methods won't be necessary.

Hope it helps!

EDIT: For further reference, I think the following link will be helpful. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6099145/

EDIT no. 2: As pointed out by the author of this question, the comments under my response are more efficient in explaining the concept than the original response itself, so I'll add my comments here.

If I go a little technical, the main reason for adjusting p-value is to control the family wise error rate or experiment wise error rate. And it happens when multiple tests or multiple comparisons are performed. That's why you don't need it when you are testing an universal hypothesis without employing any multiple comparison or testing procedure.

Suppose you want to check whether an aero plane functions properly. Surely there are methods where you can check whether the whole plane is in good condition to fly. But if you go for a thorough procedure, you would want to check it part by part. Now when you are checking each parts of the aero plane separately, and trying to infer based on them about the whole aero plane, you'll have to account for some adjustments keeping each parts in mind. Otherwise might conclude the aero plane is ready to fly when in reality its wing is not in balance with its empennage or something like that.

Answer 2

I think you are essentially correct. As you can see, the common thread in the three scenarios you describe is that there are multiple comparisons to make some overall inference. When you perform hypothesis testing with multiple comparisons, you generally end up looking at the test with the lowest p-value, because this is the one with the greatest evidence for a deviation from the null hypothesis. This means that you are effectively "optimising" over multiple tests. The null distribution of the lowest p-value is not uniformly distributed, and its distribution tends to put heavy density on lower values. Adjustment is then required to get an "overall p-value" that accounts for this and moves the null distribution back to a uniform distribution (hopefully).

Answer 3

Usually I explain it this way: when you have p-value 0.05 it basically means you are wrong 1 in 20 times. On the other hand, our main consideration with hypothesis is to accept or reject them, not probability.

Hence, when you have 20 p-vals 0.05 "on average" you'll accept 1 false positive. This is why you introduce correction. You lose some accepted hypothesis, but you get more accurate results.

This is really also about how you read results. Sometimes (ie. while struggling with results) people will say that due to explanatory characteristics of analysis they omitted correction or they would give both values. This kind of results should be treated with caution. Mostly, while testing multiple hypothesis, you should use correction.