3

I am trying to compare large normal distributions (n > 1000) to mu = 0. Altough visually the distribution is relatively close to 0 (i.e., a large proportion of it overlaps 0 and the opposite side), frequentist and Bayesian t-tests are extremely supportive of the alternative hypothesis.

set.seed(123)
p <- rnorm(4000, -0.3, 0.50)
plot(density(p))

t.test(p)
BayesFactor::ttestBF(p)

enter image description here

Are there any less sensitive alternatives? Methods that would require larger deviations to consider evidence for the alternative hypothesis?

Thanks

Dominique Makowski
  • 379
  • 2
    Why would you want to use an inferior procedure? Perhaps the difficulty is that you're not addressing your underlying problem correctly. Could you edit this post to explain what that problem is? – whuber Apr 26 '18 at 15:25
  • Remember, the test is testing that the mean of the distribution equals zero, which it doesn't, not whether the distribution has significant probabilities of both positive and negative values. – jbowman Apr 04 '23 at 17:13

2 Answers2

1

When the null hypothesis really is true, test p-values tend to have $U(0,1)$ distributions, no matter the sample size, so the test is just as unlikely to reject a true null hypothesis when the sample size is large as when the sample size is small.

Consequently, when your test comes back and tells you that the null hypothesis is false, that is a credible rejection, since the test does not give a particularly high false rejection rate.

If you are content to have the usual tests reject yet not flag tiny deviations from the null hypothesis, I see a few options.

  1. Formal equivalence testing, such as two one-sided tests (TOST), if you have some sense of how close qualifies as "close enough".

  2. Setting a low $\alpha$ level, perhaps much lower than $\alpha = 0.05$, so that small deviations that are not of practical significance do not get flagged as "significant". A danger here is that backing out the $\alpha$ requires knowledge of the variance, which you can estimate from data or predict from domain knowledge but probably do not know.

  1. Calculate a confidence or credible interval to show that the interval of plausible values contains values that are trivially different from the value under the null hypothesis.

However, your hypothesis test is doing what it should be doing. You fed it data from a distribution that has a nonzero mean, and it determined that the mean is not zero. I stand by what I wrote here about the Princess and the Pea.

Dave
  • 62,186
0

I am trying to compare large normal distributions (n > 1000) to mu = 0

...

Methods that would require larger deviations to consider evidence for the alternative hypothesis?

If a distribution with the alternative hypothesis, in your example $\mu = -0.3$, should not be considered to trigger evidence for the alternative hypothesis, then you should change your null and alternative hypothesis.

Sextus Empiricus
  • 77,915