Why are standard frequentist hypotheses so uninteresting?

Question

In almost any textbook introducing the topic of frequentist statistics, null hypotheses of the form $H_0: \mu=\mu_0$ or similar are presented (the coin is unbiased, two measurement devices have identical behavior, etc.). Classic statistical tests such as the $Z$ or $T$ tests are then based on rejecting these null hypotheses.

As I see it, these types of equality hypotheses are uninteresting for a number of reasons:

In "real life" one is always only interested in some finite accuracy $\epsilon$, meaning the hypothesis of interest is actually of the form $H_0: |\mu-\mu_0|<\epsilon$.
The equality hypothesis is a priori known to be wrong when considering continuous variables (no coin is perfectly unbiased in reality!), and as a corollary,
The fact that the null hypothesis cannot be rejected is by definition temporary, and is an artifact of not enough data. Given enough data, any type of equality hypothesis on continuous variables will be rejected in a real world use case.

So, why are these hypotheses still used, both in text books and in applications, while it is difficult to find formulas for more interesting* "real" hypotheses?

_{* e.g. a question I recently asked regarding these types of hypotheses}

This is a slight tangent from the main thrust of your question, but on the topic of biased coins, you may be interested in Gelman's paper "You Can Load a Die, But You Can't Bias a Coin" — Adrian, Nov 05 '22 at 19:27
I don't think what I am about to say will answer your question, however, forming an interval estimate (e.g. $(1-\alpha)100%$ CI) basically avoids all of the shortcomings by focusing on what $\mu$ is likely to be, rather than focusing on what it isn't — jcken, Nov 05 '22 at 20:03
@jcken, your interpretation of a CI sounds more like one of a credible interval (or perhaps a fiducial interpretation of confidence interval) than a frequentist confidence interval to me. I wonder if a particular realization of a confidence interval tells us much about what $\mu$ is likely to be. — Richard Hardy, Nov 05 '22 at 21:01
@RichardHardy you're right. I guess the point I want to make is that concentrating on estimation (and in particular, interval estimates) rather than testing would, I think, solve/avoid each of the 3 problems posed in the question — jcken, Nov 05 '22 at 21:13
This of course is a loaded question, and such questions are not usually very helpful to learn something that goes beyond already existing prejudices. — Christian Hennig, Nov 06 '22 at 01:02
@Glen_b That’s been my observation, too, that people are reluctant to pick an epsilon. — Dave, Nov 06 '22 at 04:33
@Glen_b Of course one can always compute a confidence interval and see whether something smaller than $\epsilon$ is in it, and without even specifying an $\epsilon$ one can see whether everything in the CI looks substantially big or not. — Christian Hennig, Nov 06 '22 at 09:41
@Glen_b - TOTS was exactly what I was looking for. Thank you! Do you want to add this as an answer? — Nathaniel Bubis, Nov 06 '22 at 14:49
This question seems to be in part about the falsification of scientific hypotheses. This topic has been discussed previously on CV; see for example: 1, 2, 3, 4, ... — dipetkov, Nov 06 '22 at 15:18
@nbubis If that's really an answer, the question should probably close as a duplicate since there are multiple questions on site for which my comment is already an answer. — Glen_b, Nov 06 '22 at 22:04
I'll think about how to frame an answer that doesn't seem like a duplicate. — Glen_b, Nov 06 '22 at 22:18

score 21 · Answer 1 · edited Nov 06 '22 at 13:17

21

You have to crawl before you walk, and simple examples like testing a coin for bias, under a null hypothesis of zero bias, make for a teachable example for complete beginners (which every single one of us was at one point).

Jumping straight to, say, equivalence testing without even discussing easier null hypothesis significance testing seems like poor pedagogy. Where the majority of statistics education seems to suffer is when it comes to teaching the numerous limitations of hypothesis testing. After all, the OP is right: we basically always know the null hypothesis to be at least a little bit false, and we probably are more interested in something along the lines of $H_0: \vert \mu_1-\mu_2\vert<\epsilon$. Better integration of equivalence testing like this into early statistics curricula is an interesting idea—at least after some basics are covered.

edited Nov 06 '22 at 13:17

Nick Cox

56,404
8
127
185

answered Nov 05 '22 at 21:35

Dave

62,186

1

Is it covered anywhere though? If indeed we agree that an epsilon type hypothesis should be the one practically used, you'd expect textbooks and guidance to at least mention it and state versions of standard tests for it. – Nathaniel Bubis Nov 05 '22 at 21:58
2

@nbubis Equivalence testing probably should come up more often than it does, yes, though it depends on the field where you work. If you want to show your new medication to be more effective than that of competitors, a standard test makes sense. If your big selling point is that your medication is much cheaper, then perhaps it’s enough just to show that it has efficacy equivalent to that of competitors, and then equivalence testing might be your friend. – Dave Nov 06 '22 at 04:43
I guess whether it is basically always or not depends on the area of application. I would more readily agree it is "often". – Richard Hardy Nov 06 '22 at 08:31
5

There is not just one but many versions of Catch-22 here. Often hypothesis testing has to be taught early because the students will meet it soon, because your colleagues assume that you will teach it, and so on. But it's uncomfortable for everybody to introduce a body of ideas together with vast reservations and criticisms of why it is limited and even wrong: uncomfortable for teachers, uncomfortable for students. – Nick Cox Nov 06 '22 at 13:23
2

An alternative that say the entire first course or text should focus on exploratory data analysis and ideas about measurement and research design without touching on statistical inference has sometimes been tried but usually abandoned. So far as I can tell, the introductory textbooks have mostly not changed much in some decades in the balance between descriptive and inferential statistics, apart from gimmickry such as photos of happy people, fun facts, silly quizzes, and even (no names here) sprinklings of off-colour jokes because supposedly that's what students will react to. – Nick Cox Nov 06 '22 at 13:27
1

Equivalence testing is in the end equivalent to constructing confidence intervals, which are intervals of non-rejected hypothesis tests. The problem is mostly the 'null' in null hypothesis testing but not 'hypothesis testing'. – Sextus Empiricus Nov 08 '22 at 09:06
@NickCox and others: This discussion inspired the question I have asked here: https://stats.stackexchange.com/questions/595032/should-we-teach-and-use-equivalence-tests-given-that-we-have-confidence-interval – Christian Hennig Nov 08 '22 at 11:23

Christian Hennig · Answer 2 · 2023-08-13T14:53:56.833

No model is ever true. This means that not only the null hypothesis is not true, neither is the alternative, nor something like $|\mu_1-\mu_2|<\epsilon$. If you're interested in which model is true, you're generally lost in model-based statistics; there's nothing particularly wrong about standard null hypotheses. Whether the $H_0$ or any parametric model is true is simply the wrong question.

Of course you can decide yourself what you're interested in, but I find often informative whether or not data give any evidence against a simplistic random variation "nothing meaningful is going on" model. Of course we all know that non-rejection doesn't mean the null model is true, but if you can't reject it you should really really really not claim that the data show anything meaningful, and by the way then of course you can't reject $|\mu_1-\mu_2|<\epsilon$ either, whatever $\epsilon$.

Of course you should be interested in effect sizes and not only run a null hypothesis test, so that even in case your point null is rejected you can see whether data are still compatible with a ridiculously low effect (i.e., compute a confidence interval and see whether something as small as $\epsilon$ is in it, in case you can specify a "critical $\epsilon$").

Baseline: What's "interesting" is subjective, but the rationale of testing a point null is not the question whether it's true (it isn't, but it isn't alone at that), but rather whether there's any clear "signal" in the data deviating from it. In case there is, you've got to do more to learn more.

Note particularly that it is one thing to actually reject the $H_0$, but quite another to just claim, in case you didn't reject it, that you'd have rejected it with more data. Particularly then you won't have any idea in which direction things will go. And also, D. Mayo made the valid point that if rejecting a $H_0$ were so easy indeed, why is it often so hard to replicate rejections?

Another consideration is that in fact tests will not always reject a false null hypothesis with a large enough data set, because (a) many standard frequentist tests are one sided and with a large data set things may go to the wrong side, and (b) in case the nominal model is wrong (which it always is), one can in many cases even find other models that imply a low probability to reject the null hypothesis for a range of parameters and also for big data sets, for example if you use a t-test and the true underlying distribution has very heavy tails and/or produces outliers, or if you have negative correlation between observations.

"This means that not only the null hypothesis is not true, neither is the alternative". Given that the alternative is the negation of the null, exactly one of them must be true. Arguably both are false if the mean does not exist, but that's not something that happens all the time. — Betterthan Kwora, Nov 06 '22 at 21:24
@BetterthanKwora The alternative in standard parametric testing is the same parametric model but with different parameters, for example observations i.i.d. Normal$(\mu,\sigma^2)$ with values $\mu$ different from the null hypothesis. Real observations are never normal as they are always discrete. Furthermore arguably nothing in the world is ever truly independent, or identically repeatable. De Finetti wrote that "probability does not exist" (meaning objective frequentist probabilities), and he had very good arguments for that. — Christian Hennig, Nov 06 '22 at 21:51

score 10 · Answer 3 · answered Nov 05 '22 at 21:11

10

The appeal of such hypotheses lies in their simplicity and the analytical tractability (or just simplicity) of testing them.*

In "real life" one is always only interested in some finite accuracy $\epsilon$, meaning the hypothesis of interest is actually of the form $H_0: |\mu-\mu_0|<\epsilon$.

This might be true for continuous phenomena, but not for discrete ones. E.g. in genetics a gene either has an effect on something or not. (I guess there can be even better examples than that.)

The fact that the null hypothesis cannot be rejected is by definition temporary, and is an artifact of not enough data. Given enough data, any type of equality hypothesis on continuous variables will be rejected in a real world use case.

Again, this holds for continuous phenomena only.

What we can criticize is perhaps the choice of examples in textbooks. Perhaps more examples of discrete phenomena are in order.

*Your criticism could just as well be directed at statistical models. These are often quite simple (e.g. the ubiquitous linear models), and much of their appeal is also in their tractability and simplicity of interpretation. Or even models in general, as again they are simplifications of reality with all of the drawbacks (but also benefits) that come with that.

answered Nov 05 '22 at 21:11

Richard Hardy

67,272

2

I noted this was true only for continuous models. I think the fundamental difference between point hypotheses and say linear regression is that the latter converge to "usefulness" as we get more data, but the former becomes less useful. – Nathaniel Bubis Nov 05 '22 at 21:57
1

@nbubis Linear regression does test point hypotheses. In particular, the most common test in a regression is if some regression coefficient (or several of them) is zero or not. – Dave Nov 06 '22 at 04:45
1

@nbubis, I think I can see your point. At the same time, sticking to a linear model of a nonlinear real-world phenomenon as the sample size grows often implies a failure to utilize the data better. (Think about learning curves in machine learning and their use in model selection and expansion of the candidate model set.) – Richard Hardy Nov 06 '22 at 08:37
1

The gene example may not be a good one. All genes compete for resources to an extent, at least in theory. (In practice this effect is so small as to be unmeasurable in many cases; this is not the same as saying that it does not exist.) – TLW Nov 06 '22 at 20:19
@TLW, well then, my bad. I welcome suggestions of cleaner examples! – Richard Hardy Nov 06 '22 at 20:23

Sextus Empiricus · Accepted Answer · 2022-11-08T11:27:19.657

You can regard this as an example of 'All models are wrong but some are useful'. Null hypothesis testing is a simplification.

Null hypothesis testing is often not the primary goal and instead it is more like a tool for some other goal, and it is used as an indicator of the quality of a certain result/measurement.

An experimenter wants to know the effect size and know whether the result is statistically significant.

For the latter, statistical significance, one can use a null hypothesis test (which answers the question whether the observation has a statistically significant deviation from zero).

The null hypothesis test and p-values are now considered as a bit of an old fashioned tool. Better expressions of experimental results are confidence intervals or intervals from Bayesian approaches.

The equality hypothesis is a priori known to be wrong

Yes, if you consider coins.

But an exception might be hardcore science like physics or chemistry where certain theories are tested. For instance the equivalence principle.

In addition if the equality hypothesis is a priori wrong, then why perform an experiment? If something is a priori wrong then the point is not to show that this something is wrong, instead the point is to show that there is an effect that can be easily measured. A casino that wants to test coins may not care about the theoretical probability that coins are not exactly p=0.5 fair and might differ by some theoretically small value, they care about finding out coins with a larger difference. And the point of the null hypothesis test is to prevent false positives.

Also note the two approaches/philosophies behind null hypothesis testing

Fisher: You may have observed some effect, and as expected it is not zero, but if your p-value is high then it means that your test has little precision and little strength in differentiating between different effect sizes (even down to a true effect size of zero, the observed effect may have likely occured and thus statistical fluctuations govern your observation). So you better gather some more data.

The p-value and null hypothesis is a rule of thumb for indicating precision of an experiment.
Neyman and Pearson: (from 'On the Problem of the Most Efficient Tests of Statistical Hypotheses')

Indeed, if $x$ is a continuous variable – as for example is the angular distance between two stars – then any value of $x$ is a singularity of relative probability equal to zero. We are inclined to think that as far as a particular hypothesis is concerned, no test based upon the theory of probability can by itself provide any valuable evidence of the truth or falsehood of that hypothesis.

But we may look at the purpose of tests from another view-point. Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.

The hypothesis test is a practical device to create a decision rule. One goal is to make this rule efficient and using a likelihood ratio with a null hypothesis is one method of achieving that.

In "real life" one is always only interested in some finite accuracy $\epsilon$, meaning the hypothesis of interest is actually of the form $H_0: |\mu-\mu_0|<\epsilon$.

This is captured by hypothesis testing. An example is two one-sided t-tests for equivalence testing and can be explained with the following image and can be considered as testing three hypotheses instead of two for the absolute difference

$$\begin{array}{}H_0&:& \text{|difference|} = 0\\ H_\epsilon&:& 0 <\text{|difference|} \leq \epsilon\\ H_\text{effect}&:& \epsilon < \text{|difference|} \end{array}$$

Below is a sketch of the position of the confidence interval within these 3 regions (unlike the typical sketch of TOST, there are actually 5 situations instead of 4).

The point of observations and experiments is to find a data driven answer to questions by excluding/eliminating what is (probably) not the answer (Popper's falsification).

Null hypothesis testing does this in a somewhat crude manner and does not differentiate between the situations B, C, E. However, in many situations this is not all too much of a problem. In a lot of situations the problem is not to test tiny effects with $H_0: |\mu-\mu_0|<\epsilon$. The effect size is expected to be sufficiently large and above some $\epsilon$. In many practical cases testing $|\text{difference}| > \epsilon$ is nearly the same as $|\text{difference}| > 0$ and the null hypothesis test is a simplification. It is in the modern days of large amounts of data that effect sizes of $\epsilon$ play a role in results.

Before this issue was dealt with by having arbitrary cut-off values for p-values and by power analysis. If a test had a p-values below some significance level, then the conclusion is that the effect must be some effect beyond some size. These p-values are still arbitrary, also with TOST equivalence testing. A researcher has some given significance level and computes a required sample size to obtain a given power for a given effect that the researcher wants to be able to measure. The effect of replacing $H_0$ by some range within $\epsilon$ is effectively changing the power curve. For a given effect size close to $\epsilon$ the power is reduced and it becomes less likely to reject the null hypothesis. It is effectively just a shift in the power.

Why are standard frequentist hypotheses so uninteresting?

They are simple basic examples that allow for easy computations. It is easier to work with them. But indeed, it is more difficult to imagine the practical relevance.

score 4 · Answer 5 · answered Nov 06 '22 at 13:32

(1) The more boring a null hypothesis, the more interesting it is when you are unable to reject.

E.g. after one million flips, we still cannot distinguish the coin from perfectly unbiased. (After one million patients, we still cannot distinguish the treatment from placebo.)

(2) If your question isn't a testing question, don't use a test to answer it.

E.g. instead of a yes/no question "is the coin biased", you want to estimate "how biased is the coin". (What is the impact of a disease on life expectancy.)

score 4 · Answer 6 · answered Nov 08 '22 at 02:31

4

To calculate a p-value, you need a null hypothesis such that the test statistic has a known distribution. Simple asserting $|\mu|<\epsilon$ doesn't give you a distribution.

answered Nov 08 '22 at 02:31

Acccumulation

5,206

Neither does $|\mu|=0$. – Nathaniel Bubis Mar 02 '23 at 11:09
1

@nbubis $|\mu|=0$, with other assertions, gives a distribution. For instance, with a normality assumption, this gives you a t-distribution for $\frac {\sqrt n (\bar x - \mu)}{s}$. – Acccumulation Mar 03 '23 at 00:34

score 3 · Answer 7 · answered Nov 08 '22 at 08:59

In "real life" one is always only interested in some finite accuracy ϵ, meaning the hypothesis of interest is actually of the form H0:|μ−μ0|<ϵ.

Indeed, it is an important practical issue that a significant effect turn out to be small and therefore "insignificant" for practical purposes. As an anecdotal example: bilingual children start speaking later than their peers in monolingual families... but this discrepancy is smaller than that between boys and girls (boys start speaking later than girls.) - The effect is statistically significant, but of too little importance to deprive your child from learning a second language from their birth.

There are procedures for testing whether the effect is bigger than some pre-specified value, see Equivalence testing. However these testing procedures are based on simpler testing procedures, which therefore must be learned beforehand.

The equality hypothesis is a priori known to be wrong when considering continuous variables (no coin is perfectly unbiased in reality!), and as a corollary,

The fact that the null hypothesis cannot be rejected is by definition temporary, and is an artifact of not enough data. Given enough data, any type of equality hypothesis on continuous variables will be rejected in a real world use case.

The seeming paradox arises here, because we abandon some idealizations, while preserving the other: thus, while taking into account that no coin is perfect, we still assume that a) we can make an infinite number of trials, and b) the effect is important (greater than some $\epsilon$.) A well known adagio about models is that "All models are wrong, but some are useful" and statistical testing is a case in point.

In short story On Exactitude in Science Jorge Luis Borges has given a well-known poetic description of why any useful model/theory is necessarily approximate, whereas the one that takes into account everything is totally useless. (See here for the full text.)

score 1 · Answer 8 · answered Nov 06 '22 at 23:10

I wouldn't put too much philosophical labor into contemplating the null hypothesis per se, as the OP does.

As I have discussed here, this is a device through which we can determine whether the data allow us to assert probabilistically the direction of influence.

And the direction of influence is a heavyweight, in all worlds.

Why are standard frequentist hypotheses so uninteresting?

8 Answers8

Linked