3

For example, a researcher could cleverly adjust how may hypotheses tests fall within the "family" definition in order to achieve more (or fewer) rejections when calculating the Bonferroni correction.

Follow-up question: Do false discovery rate methods such as the Benjamini–Hochberg procedure avoid this conundrum?

RobertF
  • 6,084

2 Answers2

2

False discovery rate methods don't get around the conundrum, as you state it. In general, methods for family-wise error rate control are aimed at controlling the probability of any type I error occurring across a family of hypotheses, while methods for false discovery rate control are instead designed to control the proportion of positive results that are type I errors across a family of hypotheses. Both methods rely on some a priori definition of what that "family" of hypotheses is, but have somewhat different theoretical approaches for conceptualizing "error control".

EDIT: Alexis comments below that: "False discover rate methods do not require a definition of family, and they scale rejection probabilities to different numbers of comparisons so that the FDR rate is conserved regardless of number of comparisons". It is true that FDR methods are invariant with respect to the number of comparisons, but I don't think it is strictly true that they do not require a definition of a family, but rather that they relax the stringency of the assumptions regarding that family, analagous to some of the methods I discuss later. Regardless, FDR methods conceptualize error differently than FWER methods, but still require careful consideration of what the "set" of hypotheses are.

In terms of what constitutes a family, this isn't clear cut. The most general definition would be the one HEITZ provides, which is "all of the tests that are performed." However, this definition isn't particularly clear, either; does it refer only to the set of a priori hypotheses related to the outcome of the study? Or does it refer to all hypothesis tests conducted at any stage of the study? Do studies analyzing data from publicly-available datasets have to adjust their findings for ever hypothesis tested on that dataset from other studies? The scope of these questions veer quickly out of the realm of statistical theory and into the realm of philosophy of science. This is one of the many motivations for some paradigms, for example Bayesian inference, being favored over traditional frequentist approaches, since they avoid the question entirely by not focusing on the null hypothesis testing framework (not that Bayesians aren't forced to answer such questions in different ways, for example in the assumptions made about priors, but the point is that they don't need to worry about family-wise error in the frequentist sense).

In addition, there are more rigorous and flexible ways of handling type I error than simple FWER or FDR methods like the ones you mention. For example, generalized procedures based on the closed testing principle (see [1]) or one of a variety of "gatekeeping" and graphical procedures (see [2]). Note that these do not solve the problem of how you define a "family" of hypotheses, but address the problem in a different way by allowing for a more nuanced consideration of how different combinations of null hypothesis tests may relate to one another.

[1]: Kevin S.S. Henning & Peter H. Westfall. "Closed Testing in Pharmaceutical Research: Historical and Recent Developments." Stat Biopharm Res. 2015; 7(2): 126-147. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4564263/

[2]: Mohamed Alosh, Frank Bretz, & Mohammad Huque. "Advanced multiplicity adjustment methods in clinical trials." Statistics in Medicine. 2014; 33(4): 693-713. https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.5974

Ryan Simmons
  • 1,873
  • As of this writing, the first paper is available at the link provided, but the full text of the second paper is behind a paywall (I access it through an institutional account). Unfortunately, at the moment I am not aware of a free pdf version of it. – Ryan Simmons Apr 24 '18 at 22:52
  • Thanks Ryan! As a thought experiment, say we have two sets of 1,000 uniformly distributed p-values .001, .002, ..., 0.999 where H0 = True for outcomes y1 and y2 on the same subjects, plus 50 p-values in each set < .001 where Ha = True. For alpha = .05, the Expected FDR will be ~ 0.50 whether we combine both sets together as a single family, or treat them as two separate families of tests, right? However, the Bonferroni adjusted alpha will change from .05/1,050 for each separate family of p-values to .05/2,100 for the combined family, which could mean significantly fewer or more rejections. – RobertF Apr 25 '18 at 02:18
  • 1
    False discover rate methods do not require a definition of family, and they scale rejection probabilities to different numbers of comparisons so that the FDR rate is conserved regardless of number of comparisons. – Alexis Apr 25 '18 at 23:08
  • @Alexis I would tend to disagree, though perhaps it may be more accurate to say that FDR methods relax the stringency of the definition of family. However, the methods implicitly rely on some concept of a family (in this case, the set of "discoveries" out of the total set of tested hypotheses). In fact, though it has been a while since I've read about this in any detail, FDR methods are often framed as being a particular type of family-wise error rate method, providing "weak" control (i.e. they are more liberal/anti-conservative than "strong" control methods like Bonferroni). – Ryan Simmons Apr 26 '18 at 13:02
  • @Alexis But since a really "in the weeds" discussion of how FDR and FWER relate is probably beyond the scope of this question, even if I'd be happy to have it in general. I'm amending my answer a bit in deference to your comment. – Ryan Simmons Apr 26 '18 at 13:02
  • @RyanSimmons Thank you for the thoughtful take on my edit. I agree that both FWER and FDR require a set of comparisons. However, FWER methods are—in my opinion—partly hamstrung by their rejection probabilities' strong dependence on the specific set. This sensitivity is connoted by (utterly arbitrary) "family," and that is why I would not describe FDR as including a "family." I will remove my downvote, and even upvote if you also elaborate on the relative sensitivity of FWER rejection probabilities to the definition of family. – Alexis Apr 26 '18 at 14:59
  • @Alexis Alexis & Ryan, would it be fair to say that FDR methods are (at least partially) immune to "p-hacking"? Whether or not you're dealing with 100 or 1,000 tests in your "family", if you plot out the ranked p-values as recommended in Schweder & Spjotvoll’s 1982 paper, you'll see the H0 = True tests fall along the line while Ha = True tests will tend to deviate from the line. – RobertF Apr 26 '18 at 18:21
0

There is a rigorous definition of ‘family’ - all of the tests that are performed. You are not really ‘allowed’ to get creative. You with come to the table with ‘planned’ comparisons or ‘post hoc tests’. If conducting planned comparisons, you have a set of k tests to run, defined a priori, and thus you set your family wise alpha assuming 3 comparisons. Perhaps more commonly, one uses post-hoc tests, in which case family wise alpha is set based on the number of comparisons that could, or are, made, without prior hypotheses. Tukey tests try to reduce the number of tests made by making smart comparisons; Sheffe corrections are based on all possible combinations.

You are correct - one can game the system, but you’re not supposed to.

HEITZ
  • 1,772
  • Right, but let's say you're conducting two sets of tests on separate outcomes y1 and y2 for the same subjects. Whether you decide to combine both sets of p-values together or treat them separately, won't the FDRs be the same (or the combined family FDR = the average of the two separate set FDRs), while the Bonferroni adjusted alpha could change considerably when you increase the sample size in the combined family. – RobertF Apr 25 '18 at 02:48
  • @RobertF in that case you should probably be using multivariate analysis; you still don’t really have latitude to pick what you want just to leverage some control over family wise alpha. – HEITZ Apr 25 '18 at 12:27
  • 1
    "all the tests that are performed" is not a formal definition. "All" in a single model? In a single paper? In a single data set? By a single researcher? All frequentist hypothesis tests that have ever been conducted? All frequentist hypothesis tests that ever will be conducted? All frequentist hypothesis tests that could be conducted? There's no family-wise error rate theory that defines "family." – Alexis Apr 25 '18 at 23:05