11

It's been my understanding that the null hypothesis is never or rarely really true.

Then, isn't the real point of statistical testing to detect if there's an effect size large enough for the test to show statistical significance at a given $\alpha$ level? (edit: I mean an effect size one is interested in, which depends on one's research question).

If I'm correct, doesn't it mean that ideally we should always conduct some a priori power analysis? Or is my reasoning somehow incorrect? Are there situations where a priori power analysis isn't really necessary?

I don't have a specific situation or problem in mind, that's why my question is really general.

Alexis
  • 29,850
Mektia
  • 113
  • 5
  • The null hypothesis, in its two tailed form. One tailed, the null hypothesis can be false. – Jeremy Miles May 01 '23 at 07:44
  • 1
    Null hypotheses are often true. Think of composite null hypotheses. For example, the null could be that new drug B is not more effective than old drug A, while the alternative would be that drug B is more effective than drug A. – Graham Bornholt May 01 '23 at 10:19
  • 5
    It is extremely difficult to articulate any empirical assertion (Kantian synthetic a posteriori) that is definitely true. From that perspective, then, what's the point of doing any kind of scientific work at all?? ;-) – whuber May 01 '23 at 18:01
  • 2
    Relevance tests (combined inference from "positivist" null hypotheses of "no effect" with inference from "negatvist" null hypotheses of "effect at least so big a priori") directly incorporate both effect size and statistical power into the conclusions (see the table in my answer). – Alexis May 01 '23 at 19:02

4 Answers4

5

If we focus on medical research; performing a study involves taking a risk and potentially harming people. This is acceptable within bounds defined by the Principle of Equipoise as outlined in the Declaration of Helsinki. Prior to recruiting even a single subject to a study, the protocol must be reviewed and approved by an ethics board, usually an institutional review board (IRB). Many medical centers include a statistician or epidemiologist on such boards, and they consider the statistical feasibility of the study. That is to say, the protocol statistician has outlined the assumptions and the anticipated effects and applied the necessary formulas to provide rationale for the specified sample size(s). There are a number of questions to consider subsequently: are the assumptions reasonable? Is the analysis well powered? Does it make sense to recruit this many people without additional preliminary research? Will the potential benefits in the population after the study outweigh the risks in the study participants? And so on...

The constitution and mission of an IRB is outlined in the Belmont report. Just a plug, IRBs within medical institutions often have difficulty recruiting and retaining statisticians. If you are a biostatistician within an academic medical center, ask whether there is a seat for a biostatistician to participate.

The result of a successful medical trial is that standard practice can be updated based on what is known. Typically, this does fall down to a trial showing a significant result. One can hope based on the input of IRBs, and the natural limitation of cost, that the design feature under study has a reasonable profound impact on health so that the significance is compelling in its own right.

There is a flipside to this. Much less can be said of non-experimental, large EHR based studies which often show significant effects that can't and shouldn't be translated into practice. Open data sources and semi-closed data sources often do not have a steering committee to review the ethics of proposed research. Conversely, many languishing areas of healthcare continue to hem and haw over results due to the failure of trials to show unequivocal results, such as sodium reduction, cognitive behavioral therapy, fish oil supplementation, low fat diets, some vaccines, and so on.

In summary, for any confirmatory study, no there is no point to conducting a hypothesis test unless a power/sample size calculation has been performed - and the primary endpoint(s) is/are formally powered and secondary endpoints are reasonably powerful or important. In any other case, the analysis should be treated as exploratory, and a "hypothesis test" in this framework can be viewed as yet another method to identify research topics or detect effects - in that case, the statistician should be completely transparent in the reporting of their results.

AdamO
  • 62,637
4

I'm not sure how a power analysis would help. If you do a power analysis, which says you need N = 1000 and it is not significant, what do you know that you didn't know before you did the power analysis? (And how do you estimate the size of the effect to put into the power analysis?)

This is the problem with over-relying on (or perhaps over-interpreting) p-values. A non-significant p-value tells you that you do not have confidence in knowing the direction of the effect.

Andrew Gelman, in his blog, has popularized the idea of Type S and Type M errors instead of Type I and Type II errors. A Type S error is a Sign error - you have the wrong direction of effect, a type M error is a Magnitude error - you have not correctly estimated the magnitude of the effect.

Jeremy Miles
  • 17,812
  • 5
    "what do you know that you didn't know before you did the power analysis?" From what I understand about power analysis, I'd know that the population probably doesn't have the effect size I'm interested in. But happy to be corrected! – Mektia May 01 '23 at 17:17
  • A power analysis tells you whether absence of evidence may be construed as evidence of absence or not. I disagree with the statement that "a non-significant p-value tells you that you do not have confidence in knowing the direction of the effect" - if you run a very well-powered study with the alternative hypothesis that A>B, you would have very likely rejected the null if A actually were greater than B. If we don't have evidence that A>B under conditions where we would have almost certainly identified it if it were true, it's reasonable to conclude that B>=A. – Nuclear Hoagie May 01 '23 at 19:50
  • 1
    Hmm... a power analysis is an appropriate setup for a confirmatory study. Hopefully prior to that, an exploratory or early phase study has given us some reasonable estimates of effect. Yes a statistician has to do some fill-in-the-blank. In the end, a well powered study with a significant result is widely interpreted as compelling and a success. As my professor said, "Clinical trials are just like playing the slots." – AdamO May 01 '23 at 20:53
4

If the null hypothesis is never really true, is there a point to using a statistical test?

If we already know that the null hypothesis is not true, then the point is not to proof that the null hypothesis is not true.

The point of the null hypothesis test is to show that a test is sensitive enough to be able to exclude certain hypothesis. The quality of a test is not the ability to show which values are most likely true, but instead it is the ability to show which values are likely not true and to show it with a large significance.

In addition, there are some issues with continuous distributions having zero probability for any specific value. So no value is ever true when we consider a continuous distribution for some parameter. Still, relevant are the distribution densities and whether the region around certain hypothesis, like the null hypothesis, are low or not.

Sextus Empiricus
  • 77,915
3

Yes! If we have done a priori power calculations to figure out the sample size we'd need to consistently detect an effect of the size we care about, and we've actually collected that amount of data, then a significant p-value is confirmatory and meaningful. You made a deliberate effort to collect enough data to rule out the straw-man argument of "What if your results are just sampling variation?" and you appear to have overcome that hurdle. In Deborah Mayo's words, you subjected your hypothesis to "severe testing," using a test with "an overwhelmingly good chance of revealing the presence of a specific error, if it exists --- but not otherwise."

But if we haven't done a priori power calculations, and we chose a sample size in other ways (convenience, or budget constraints, or a mistaken belief that "n=30 is big enough" for everything)... then our test was not "severe.".

So, what's the use of hypothesis testing without an a priori power analysis? Sometimes we're in a situation where we simply couldn't have collected more data. (Maybe we are looking back at historical records and there's only a small sample left in existence. The population was larger than this sample, but there's no way to sample more data from that population any longer.) Then hypothesis testing isn't ideal, but might still be useful in a limited way: Although a significant p-value wouldn't tell us much, an insignificant p-value would tell us that we definitely should be worried about sampling variation as we interpret our findings.

civilstat
  • 3,785
  • 1
    I think you have your last point backwards. In the absence of a power analysis, an insignificant p-value is fairly meaningless, as it's impossible to discern whether it's a case of having lots of data showing a true effect size arbitrarily close to zero, or a case of having very little data failing to rule out a very large effect size. I don't see how one can interpret an insignificant p-value without a sense of the power of the study. – Nuclear Hoagie May 01 '23 at 20:42
  • I agree that you wouldn't be able to use an insignificant p-value to distinguish between those two options if you don't know the power. But I don't think you should be trying to do so, even if you do know the study has high power. Instead of hoping for a p-value to be informative about the effect size, I'm just saying an insignificant p-value is informative about the sample size: "Whatever the effect size is, your sample's too small to measure it precisely." – civilstat May 02 '23 at 01:10
  • I find that last statement an odd way to think about it, an insignificant p-value does not suggest that you "didn't collect enough data" - you may have plenty of data to convincingly rule out effects. It may not be that the sample is too small to find an effect that you actually care about, it may be that the sample is sufficient to show the absence of effect sizes you care about. Your claim that the N=30 test isn't "severe" seems to use some implicit sample size calculation to suggest it's not "enough", but if you only care about large effect sizes, N=30 may plenty. – Nuclear Hoagie May 02 '23 at 13:27
  • If your question of interest is about the effect size, wouldn't you use a confidence interval instead of a hypothesis test? – civilstat May 02 '23 at 15:46