4

Suppose we are in the following setting. There are 10 covariates and 1 response along with some other data derived from those covariates.

One can do regression of 10 covariates to 1 response to test individual coefficients significance.(This constitutes 10 tests though they might be dependent upon each other.) One can do testing on data derived from the covariates. Say 10 different tests have been done.

If I am using $\alpha=5\%$, am I going to run into the issue of inflating type I error here? I would say expected number of type I error is upper bounded by 2 out of 20 tests total due to correlation. However, for $\alpha=1\%$, I think problem would be alleviated a bit here.

How many test can one do in medical research article? Similarly, how many tests can be done on the same data set assuming a thorough exploratory data analysis has been done?

user45765
  • 1,416

2 Answers2

3

The whole multiple comparisons question has been discussed here many times. See e.g. here, here, and here, and the whole mutltiple-comparisons tag. Then there's the whole question of the utility of hypothesis tests and p values in the first place, also discussed here many times, e.g. here, and here (and all those threads link to other threads).

As to your other question on how many tests in a medical research article, while an individual journal may have some limit (although I don't know of any that have explicit limits), in general, as many as needed.

Finally, in one of my favorite statistics books, Statistics as Principled Argument, Robert Abelson argues strongly against statistical "rules" of this sort and instead argues that analysis should be base on what can be justified in any particular situation.

Peter Flom
  • 119,535
  • 36
  • 175
  • 383
-2

Indeed, any time you report a table of p-values for either separate test results or regression coefficients based on a Type I error of $\alpha=0.05$, you should take into account the multiple comparisons problem. One of the most conservative methods to account for this error is to use the Bonferroni correction, for which $\alpha^*=0.05/\#\mathrm{tests}$. Hence, p-values need to be less than $\alpha^*$ to be considered significant. However, Bonferroni is very conservative. Another approach to correct for family-wise error rate is to employ the Benjamini-Hochberg (1995) false discovery rate (FDR) to the p-values. Another approach involves calculation of Storey q-values to address the positive FDR (pFDR). Lastly, the Westfall-Young method is another approach which is informative.

  • This does not answer the question, and the suggestion that "any time you report a table of p-values" you have to deal with multiple comparisons adjustments is just false. Please research the topic yourself on this site. Perhaps start with the links provided in Peter Flom's answer. – Michael Lew Dec 15 '23 at 20:19