Hypothesis testing: Fisher vs. Popper vs. Bayes

Question

I try to make my question short. I am familiar with Popper’s philosophy as well as with statistical hypothesis testing after Fisher and Neyman-Pearson. I am not so familiar with the Bayesian approach except that this methodology is defined by including prior probabilities which will influence posterior probabilities. Very often, I find that null hypothesis testing is being linked to Popper although it is the result of the works of Fisher and Neyman-Pearson. Also, if I understood Popper’s version of hypothesis testing correctly, he says that one should sincerely try to disproof hypotheses – and I am quite certain that he didn’t mean the null hypothesis that Fisher formulated but rather the hypothesis that is of critical importance to us. Besides that, I think the timing when Fisher introduced the null hypothesis was a few years before Popper formulated his account on hypothesis testing.

Question 1: If my understanding is correct, isn’t this abstract partly incorrect?

J Sports Sci. 2013;31(9):919-20. doi: 10.1080/02640414.2012.753636. Epub 2012 Dec 19.

Testing the null hypothesis: the forgotten legacy of Karl Popper?

Wilkinson M.

Abstract Testing of the null hypothesis is a fundamental aspect of the scientific method and has its basis in the falsification theory of Karl Popper. Null hypothesis testing makes use of deductive reasoning to ensure that the truth of conclusions is irrefutable. In contrast, attempting to demonstrate the new facts on the basis of testing the experimental or research hypothesis makes use of inductive reasoning and is prone to the problem of the Uniformity of Nature assumption described by David Hume in the eighteenth century. Despite this issue and the well documented solution provided by Popper's falsification theory, the majority of publications are still written such that they suggest the research hypothesis is being tested. This is contrary to accepted scientific convention and possibly highlights a poor understanding of the application of conventional significance-based data analysis approaches. Our work should remain driven by conjecture and attempted falsification such that it is always the null hypothesis that is tested. The write up of our studies should make it clear that we are indeed testing the null hypothesis and conforming to the established and accepted philosophical conventions of the scientific method.

Question 2: Is there any record on what Popper's position was regarding null hypothesis testing (after Fisher and Neyman-Pearson) and the Bayesian approach as tools to gain knowledge in the sciences?

Question 3: Are there any records of communication between Popper and Fisher regarding hypothesis testing?

Answer 1

According to Mayo, Popper did not designate statistical tests implementing his logic of falsification, or as Hilborn and Mangel put it "Popper supplied the philosophy, and Fisher, Neyman and colleagues supplied the statistics", see references in Quinn and Keough's Experimental Design and Data Analysis for Biologists (Ch. 3). Popper viewed probability somewhat dismissively because "as scientists, we do not seek highly probable theories, but explanations; that is to say, powerful and improbable theories". Nonetheless, the corroboration measure he proposed in 1954 is surprisingly Bayesian (surprisingly because Bayesianism is usually associated with induction Popper rejected).

Fisher emphasized that we can never prove a theory, but only refute it, already back in 1925, before Popper's Logik der Forschung (1934), but he implemented this principle in a rather peculiar way. Instead of testing the actual hypothesis directly Fisher suggested testing its "negation", the null hypothesis, if the null hypothesis comes out as unlikely after the test then our actual hypothesis "survived the test".

Why such a detour? Because Fisher's approach can not directly provide what a scientist actually wants, $p(H|D)$, the probability of the hypothesis given the test data. It only provides the converse $p(D|H)$, the probability of data given the hypothesis is true, this is the famous $p$-value, a.k.a. statistical significance. In his 1952 criticism Neyman framed Fisherian testing as follows, if $p(D|H)$ is low "what we actually observed would be a miracle. We don't believe in miracles nowadays and therefore we do not believe in H being true". Unfortunately, a simple computation shows that $p(H|D)=\frac{p(D|H)}{p(D)}p(H)$, and the relation of $p$-value to the probability of interest depends on the prior probability $p(H)$, a.k.a. the "base rate", and reliability of data $p(D)$.

Fisher was aware that his "no miracles" argument for the null hypothesis significance testing (NHST) may break down in some circumstances (see also the base rate fallacy), especially when low probability hypotheses are involved, and emphasized the importance of choosing the right kind of hypotheses.

However, a different type of significance testing involving the null and the "alternative" hypothesis, proposed by Neyman and Pearson, is arguably closer to "sophisticated falsificationism" of Lakatos than to Popper's own. Lakatos differs from Popper on two key issues: he allows auxiliary hypotheses to "save" a theory in view of a "falsifying" test, and he encourages keeping a "falsified" theory until a clear alternative emerges. Accordingly, the Neyman-Pearson procedure presupposes adjudication between the null and the "alternative" hypothesis, the one of interest, unlike Fisherian testing that delivers a (potentially flawed) up or down verdict. As with Popper, Neyman and Pearson developed their procedure well before Lakatos formulated his methodology. Despite their differences, Fisher and Neyman more or less came to an understanding in 1970s, and both types of significance testing became "the standard procedure".

More recently, there has been something of a revolt against significance testing of both types, especially in soft sciences. According to Prevost (p.25), in 1999

"the APA Task Force on Statistical Inference seriously thought about banishing tests of significance from all APA journals... Even though they decided not to do it, they still recommended a reduction of the use of tests of significance et suggested to use of more useful methods (e.g. effect sizes, confidence intervals, Bayesian estimations,...)" [APA is American Psychological Association].

A frequent criticism is that $p$-values have little actual "significance" to decision making about research, and in particular, that significance testing does not really implement the "logic of falsification". See Nickerson and Giere for opposing points of view. The root of the problem seems to be that $p(H|D)$ can not be directly computed, so direct implementation of Popper's philosophy in probabilistic contexts may not be feasible. Bayesian inference is commonly suggested as an alternative.

Wilkinson overreaches in his identification of NHST with Popper's falsification, and his critique of induction methods depends on this overreach. The traditional position is more diffused. It suggests that NHST encourages formulation of falsifiable hypotheses and evaluates them based on test data, not that it represents the statistical implementation of Popper's criterion. Fisher's deductive reasoning gets one from the null hypothesis to the $p$-value. But interpreting the latter to judge the actual hypothesis, and formulating the null hypothesis itself, depend on pragmatic considerations that are, in essence, inductive. A Bayesian would object that in her approach these pragmatic considerations are, at least, quantified, through assignment of prior probabilities, etc., whereas in NHST they are hidden from view. While there are good arguments for preferring NHST to Bayesian inference, Wilkinson's is not one of them.

Answer 2

Into Popper's The Logic of Scientific Discovery (1934 - 1st Engl.ed.1959) you can find CH.VIII : Probability and several Appendces devoted to probability theory.

Bayes [Bayes,Th. 144, 168n, 288–9] and Fisher [Fisher,R.A. 18&n, 326, 334, 384, 394n, 403] are presents into the Index of names, also if Keynes and von Mises are prominent.

The basic "axiom" of Popper's initial view is that probability cannot solve the Problem of Induction :

I do not think that it is possible to produce a satisfactory theory of what is traditionally called ‘induction’. On the contrary, I believe that any such theory — whether it uses classical logic or a probability logic — must for purely logical reasons either lead to an infinite regress, or operate with an aprioristic principle of induction, a synthetic principle which cannot be empirically tested.

Scientific theories can never be ‘justified’, or verified. But in spite of this, a hypothesis $A$ can under certain circumstances achieve more than a hypothesis $B$ — perhaps because $B$ is contradicted by certain results of observations, and therefore ‘falsified’ by them, whereas $A$ is not falsified; or perhaps because a greater number of predictions can be derived with the help of $A$ than with the help of $B$. The best we can say of a hypothesis is that up to now it has been able to show its worth, and that it has been more successful than other hypotheses although, in principle, it can never be justified, verified, or even shown to be probable. This appraisal of the hypothesis relies solely upon deductive consequences (predictions) which may be drawn from the hypothesis. There is no need even to mention induction [ page 316-17 ].

Answer 3

Please allow a correction: I think your statement "he says that one should sincerely try to disproof hypotheses – and I am quite certain that he didn’t mean the null hypothesis that Fisher formulated but rather the hypothesis that is of critical importance to us" is not really correct. Actually, this is exactly how Fisher would have described "null hypothesis". For Fisher, there is only one type of hypothesis: the hypothesis to be tested, a.k.a. the null hypothesis. The distinction between null and alternative hypothesis, introduced by Neyman and Pearson, has never been accepted by Fisher (also Conifold's statement "Instead of testing the actual hypothesis directly Fisher suggested testing its 'negation', the null hypothesis, if the null hypothesis comes out as unlikely after the test then our actual hypothesis 'survived the test'" does not describe Fisher's approach correctly)

If you read Fisher carefully, the commonly used version of "null hypothesis" (nil, no effect, "just random variation" and similar) is not what Fisher was referring to. The central meaning of a test for statistical significance is its ability to create results which can be, following Fisher ("The Design of Experiments", 2nd ed., 1937), divided into two classes with different interpretations:

“those which show a significant discrepancy from a certain hypothesis; […] and […] results which show no significant discrepancy from this hypothesis. This hypothesis […] is again characteristic of all experimentation. […] [W]e may speak of this hypothesis as the “null hypothesis”, and it should be noted that [it] is never proved or established, but is possibly disproved, in the course of experimentation. […] If it were asserted that the subject would never be wrong in her judgements we should again have an exact hypothesis, and it is easy to see that this hypothesis could be disproved by a single failure, but could never be proved by any finite amount of experimentation. It is evident that the null hypothesis must be exact, that is free from vagueness and ambiguity, because it must supply the basis of the ‘problem of distribution,’ of which the test of significance is the solution. A null hypothesis may, indeed, contain arbitrary elements, and in more complicated cases often does so.” (p.18-20; emphasis added)

Please note that the only distinctive feature of a ‘null hypothesis’, as it is characterized by Fisher, is its ability to partition the results of an experiment into two mutually exclusive classes (supporting and contradicting cases), and to do so, it needs to be exact. That's all. It is not assumed that a null hypothesis has to state the absence of any effect, as most statistical resources – websites, articles and textbooks – claim. It is also important that, as already mentioned, in Fisher's approach there is no "alternative" hypothesis, which is frequently equated with the "research hypothesis" to be supported by rejecting the null. In Fisher's approach, the "null hypothesis" is the hypothesis the researcher wants to test. It can state the absence of an effect or its existence, and as long as it is exact, it can be tested. However, to test a non-nil null hypothesis appropriately, the usually applied tests (e.g. t-Tests) have to be replaced by versions that reflect the effect (size). In the case of the t-Test, any effect could be tested using a particular noncentral version of this test (which uses the same distribution that is used for power calculations in the Neyman-Pearson approach). A significant p would - as usual - indicate that the data obviously do not correspond with the prediction based on the null hypothesis (our research hypothesis!), which would commonly be considered as rejection of the null. Interpreted this way, at least the majority (if not all) of the usually discussed flaws of "NHST" disappear. Moreover, this interpretation of significance testing looks like a statistical version of Popper's falsification principle, or at least as a statistical argument closely related to it.

The confusion of many users of statistical methods (which has also driven the - still ongoing - discussion about NHST, "null hypothesis significance testing" in the "soft" or - better - weak "sciences" [like Psychology]) is probably due to mixing up two distinct approaches to testing hypotheses - Fisher's significance test on one side and Neyman-Pearson's theory of statistical decision on the other - into an "inconsistent hybrid that every decent statistician would reject" (Gigerenzer, 1993). A prototypical study, at least in Psychology, works like this: The researcher has an assumption, call it A, that there is some effect. He/she assumes a medium effect size (Cohen's d=0.5; poor theory, probably, but anyway...) and calculates the sample size for this assumed effect to be indicated sensitively, say with Power=0.8 (this is a kind of Neyman-Pearson). Then he/she collects data, runs a standard (central) t- or F-Test, setting a strawman null hypothesis of "no effect" and if p<0.05, he/she rejects the null (which is ok; this is Fisher, but not testing the actual hypothesis) and accepts A (which is not ok, as any effect size different from null gets support from rejecting the null, unless all other alternatives can be ruled out). This latter conclusion is neither Fisher, nor Neyman-Pearson, it is simply not correct.

To sum up, I think that the abstract you are referring to is rather one of the rare cases in which the essence of testing hypotheses (in a way that would have been accepted by Fisher) has been extracted more or less correctly. Still, there is some inexactness in the abstract, since it assumes that the null and the research hypothesis are different things. But actually, for both, Popper and Fisher, it is the research hypothesis that needs to be tested and, if necessary, rejected. Which actually is Fisher's null hypothesis.

With regard to your third question: In "Logik der Forschung" Popper sometimes refers to Fisher's likelihood concept, but not to significance testing (or to the Neyman-Pearson theory). In Bennett's (1990) "Statistical Inference and Analysis - Selected Correspondence of R.A. Fisher", Popper is not listed as correspondent.