Hypothesis testing: Calculate beta without PDF of H1

Question

I've found pages that show how to mechanically calculate beta as 1 minus alpha and/or show beta in the context of a PDF for the alternative hypothesis (H1 or Ha). If you don't have a conjectured mean for H1, however, and/or don't have a PDF (gaussian or otherwise), how does one even conceptually approach the calculation of beta?

For example, suppose Ho was "The mean of the population is m" and H1 was "The mean of the population is not m". There is no PDF for H1.

Annex: Inapplicable sources on calculating beta

some relevant answers -

https://stats.stackexchange.com/questions/135883/why-is-the-complement-to-power-not-alpha/135907#135907

https://stats.stackexchange.com/questions/397874/computing-the-power-of-welchs-t-test/397958#397958

https://stats.stackexchange.com/questions/437186/plotting-the-power-curve-for-a-sign-test/437194#437194

https://stats.stackexchange.com/questions/283927/resolving-power-in-an-ab-test/283935#283935 5. https://stats.stackexchange.com/questions/152508/how-to-compute-shapiro-wilk-test-power/ — Glen_b, Aug 07 '22 at 22:35

score 1 · Answer 1 · answered Aug 07 '22 at 21:22

1

The power of a hypothesis test (beta) is a function of effect size (and SEM, and sample size, and test 'size' (alpha). Therefore to calculate power you need to choose values for all of those variables. The choices can be thoughtful (e.g. use estimates from preliminary experimentation) or arbitrary (e.g. use alpha=0.05 just because).

Given those choices, your example of not knowing the relevant distribution is ill-directed because once you make the choices and the relevant distribution will often become known. In the case you give of the test hypothesis being that the population mean=$m$, the alternative becomes that the mean is a specific chosen value rather than just 'not $m$.

answered Aug 07 '22 at 21:22

Michael Lew

15,102

Thank you, Michael. Would you be able to paint a simple hypothetical picture of how the distribution of the mean for H1 is arrived at? Say, for example, that we are considering the North American population, and that H0 is that people are 5 feet tall, on average. H1 is that people are not 5 feet tall, on average. Are you saying that H1 cannot be so open ended, and that we must posit a specific average height for H1? If so, that is a very significant condition for the use of the concept of beta, one that I wish was more explicit on explanations found online. – user2153235 Aug 08 '22 at 06:33
The alternative hypothesis that you need is the one that comes with the settings that you choose for the power analysis. The effect size that you select specifies the hypothetical true mean. Yes, that does correspond to a "specific average height" for the power analysis, but that is not the same as the alternative that pertains when you apply the test to real data. In use the test has an alternative that is typically the complement of the null hypothesis. – Michael Lew Aug 08 '22 at 20:36
It is worth noting that the power (beta) is a factor in experimental design, but the design beta is not directly relevant to the test when it is used. The design considerations are distinct from post-test considerations. – Michael Lew Aug 08 '22 at 20:39
Thanks, Michael. Is there an online tutorial example of how a specific H1 is arrived at (from an open-ended H1) ? If not, no worries. I ran into the need to better understand beta beyond the definitional sense in trying to understand an answer to another question. I'm trying to get to a point where I don't have to dig yet another layer deeper in order to understand the preceding layer. – user2153235 Aug 08 '22 at 23:17

score 0 · Answer 2 · answered Aug 07 '22 at 20:51

Even if everything is Gaussian, you don't know the distribution under the alternative hypothesis. You pick the difference you want to be able to detect.

If you want to detect a large difference, you have a lot more power to detect it than if you want to detect a small difference; it's easier to catch that something is off my a meter than off my a nanometer.

This idea applies even when you don't have the ideal distribution (often, but not necessarily, Gaussian). In many cases, a decent sample size leads to near-convergence of the true test statistic distribution to the theorized test statistic distribution. In such a case, you might just move past the fact that your data lack the ideal distribution and just do the power calculation the usual way, knowing that your imperfet calculation will be close. If you have greater concerns, you might turn to simulation.

Are you saying that in order to use the concept of beta, H1 cannot be open ended, like "the population mean height is not $m$", where $m$ is the population mean height posited by H0? — user2153235, Aug 08 '22 at 06:34

Hypothesis testing: Calculate beta without PDF of H1

2 Answers2