11

Could someone explain why Richard McElreath says Fisher's exact test is rarely used appropriately in his excellent Bayesian introduction book (Statistical Rethinking)?

For reference, the context is below:

Why aren’t the tests enough for innovative research? The classical procedures of introductory statistics tend to be inflexible and fragile. By inflexible, I mean that they have very limited ways to adapt to unique research contexts. By fragile, I mean that they fail in unpredictable ways when applied to new contexts. This matters, because at the boundaries of most sciences, it is hardly ever clear which procedure is appropriate. None of the traditional golems has been evaluated in novel research settings, and so it can be hard to choose one and then to understand how it behaves. A good example is Fisher’s exact test, which applies (exactly) to an extremely narrow empirical context, but is regularly used whenever cell counts are small. I have personally read hundreds of uses of Fisher’s exact test in scientific journals, but aside from Fisher’s original use of it, I have never seen it used appropriately. Even a procedure like ordinary linear regression, which is quite flexible in many ways, being able to encode a large diversity of interesting hypotheses, is sometimes fragile. For example, if there is substantial measurement error on prediction variables, then the procedure can fail in spectacular ways. But more importantly, it is nearly always possible to do better than ordinary linear regression, largely because of a phenomenon known as overfitting.

Scortchi - Reinstate Monica
  • 29,907
hanice
  • 111

1 Answers1

16

It's hard to read this quotation & not surmise that the author considers it a mere blunder to use Fisher's Exact Test when the marginal totals of a contingency table are not fixed by design. "Fisher's original use" of the test must refer to the famous lady tasting tea who "has been told in advance of what the test will consist, namely that she will be asked to taste eight cups, that these shall be four of each kind, [...]" (Fisher, 1935a, Ch. 2, "The principles of experimention, illustrated by a psycho-physical experiment");† & then "an extremely narrow empirical context" parses as "a sampling scheme applicable to few studies carried out in practice".

But it's not a blunder: conditioning on the sufficient statistic for the distribution of the data under the null hypothesis is a standard technique to eliminate nuisance parameters & come up with tests having the correct significance level (unconditionally too)—it's the whole basis of permutation tests, for example. (See Neyman structure.) The marginal totals of a two-by-two table contain very little information which you can use to estimate the parameter of interest, the odds ratio; & rather a lot about the precision with which you can estimate it: the argument is that the sample space obtained by conditioning on both is much more relevant for inference than that obtained by conditioning on one only, or on the total count only. It is a horribly coarse sample space, however, when the totals are low; resulting in a lamentable loss of power. How should relevance of the sample space be balanced against information loss? How much coarsening of the sample space is acceptable before an asymptotically valid or an unconditional test is preferred? These are vexed questions, & the analysis of two-by-two contingency tables has been controversial for half a century or more.

Given that this comes from a Bayesian text, I think the author's missed an opportunity to poke fun at the dilemmas a commitment to the use of frequentist methods can lead to—like Jaynes (2012), Ch. 8, "Sufficency, ancillarity, & all that".

† In a paper published the same year as his book, he used an example (Fisher, 1935b, pp. 48 – 51) in which, though the sampling scheme is not explicitly given, at most one margin could have been fixed in advance, & most likely just the total count was fixed. Like-sex twins of convicted criminals are categorized as monozygotic vs dizygotic & as convicted of crimes themselves vs not convicted in a two-by-two table:

Let us blot out the contents of the table, leaving only the marginal frequencies. If it be admitted that these marginal frequencies by themselves supply no information on the point at issue, namely, as to the proportionality of the frequencies in the body of the table, we may recognize the information they supply as wholly ancillary; & therefore recognize that we are concerned only with the relative probabilities of occurrence of the different ways in which the table can be filled in, subject to these marginal frequencies.

[Edit: The data come from Lange (1929). Wetzell (2000), p. 162, describes Lange's data collection procedure; it's indeed the total count that was fixed by the study design.]

Fisher (1935a), The Design of Experiments

Fisher (1935), "The Logic of Inductive Inference", J. R. Stat. Soc., 98, 12

Jaynes (2012), Probability Theory: The Logic of Science

Lange (1929), Verbrechen als Schicksal: Studien am kriminellen Zwillingen

Wetzell (2000), Inventing the Criminal: A History of German Criminology, 1880 – 1945

Scortchi - Reinstate Monica
  • 29,907