5

I have samples from two random variables, and I'm assuming normal distribution with equal variance. I want to show that the distributions have equal means $\mu_1$ and $\mu_2$.

The usual two-tailed t-test is used with the inverse situation. That is, one constructs the following and hopes for a resulting $p$ small enough to reject H0.

  • H0: the null hypothesis $\mu_1 = \mu_2$
  • H1: the alternative hypothesis $\mu_1 \neq \mu_2$

But I want to reject H1, and show with statistical significance that H0 is true.

Can I use the same two-tailed t-test and just interpret the results differently? Or do I need an entirely different hypothesis test?

My first reaction was to use the same test, end up with a large $p$ instead of a small one, and then claim that H1 was rejected with confidence $1 - p$. But after messing around with some data, I'm not sure that's the right approach.

Silverfish
  • 23,353
  • 27
  • 103
  • 201
kdbanman
  • 857
  • 1
  • 8
  • 15
  • 2
    Does this answer your question? TOST and its two null hypotheses $//$ What will be the null hypothesis in this study? is good, too (maybe the real duplicate). – Dave Jan 31 '24 at 23:50
  • 1
    Yes, equivalence testing appears to be the problem setting I'm in! And TOST looks like the right test. Thank you for the links. – kdbanman Feb 01 '24 at 00:27
  • 2
    I voted to leave open: I think this question, with its more general framing, would be a better target for future (near-enough) duplicates wondering the same thing - basically, someone who's realised they might need something that turns out to be TOST despite not having heard of TOST before. To which a useful answer might be an overview of the TOST procedure, why we need the $\delta$ etc. The related questions @Dave flagged up are definitely near-dupes but feel like worse targets due to being more specific or showing more background knowledge. This Q also has an excellent, v. searchable, title! – Silverfish Feb 01 '24 at 00:41
  • 2
    Another point in this Q's favour as an independent question: being open-ended, it would also be a good home for answers that prevent alternatives or frame challenges to TOST (e.g. is the null hypothesis significance testing framework really the only or the most appropriate approach for dealing with the underlying practical problem here?) in a way that a "how do I implement TOST?" question doesn't. (Nevertheless I have edited the Q to include the [tost] tag as this may be useful to future searchers.) – Silverfish Feb 01 '24 at 00:47
  • 2
    Proper science is about investigating things with an open mind. People who "hope" or "want" certain results shouldn't be trusted as chances are they will make too much effort to achieve the result they are hoping for. By the way no model in statistics is ever literally true ("all models are wrong but..."), so there is no way to show that either null or alternative hypothesis are true, ever. – Christian Hennig Feb 01 '24 at 00:58
  • @Silverfish Retracted. Perhaps this can be a canonical duplicate (maybe even for my profile links). – Dave Feb 01 '24 at 00:58
  • 1
    @ChristianHennig The fact that "true" means of two DGPs/populations would seldom be exactly equal raises important questions about NHST - or what people are trying to achieve, really, when they resort to NHST - worth incorporating in an answer. Certainly some would rather we focus on "how large might such a difference be, judging from the data?" than try sticking a p-value on it. (Re title: your point's clearly correct but I took the intended meaning of "want" less literally, "it'd be more interesting to be able to say if evidence suggests the means are identical than that they're different") – Silverfish Feb 01 '24 at 01:55
  • 1
    @Silverfish I don't disagree with you, this is an interesting issue in its own right that deserves more than just a comment. I have written about this ;-) see here: https://arxiv.org/abs/2007.05748 particularly Sec. 3.4. – Christian Hennig Feb 01 '24 at 14:47
  • @Dave since there is interest in leaving this question open and treating it as the canonical TOST question, you can copypaste your answer here and I will accept it. (Though I'll continue to accept Ben's answer in spirit.) – kdbanman Feb 05 '24 at 00:38

1 Answers1

1

Here you can employ theology instead of statistics

If you want to accept a particular hypothesis, just accept it --- avoid wasting time by employing statistical analysis and pretending that you are interested in knowing about reality. Instead simply state your pre-empted conclusion without evidence and tell your reader that you really want this hypothesis to be true, or tell them it was an insight to you from God, etc., and that you are going to proceed accordingly. This is far more honest than trying to employ statistical analysis in a way that is constructed to try to get a pre-emptive conclusion.

Ben
  • 124,856
  • 1
    I realize it looks like I'm trying to "find the statistics to fit my conclusion." But I assure you I'm just being lazy with my writing. – kdbanman Feb 01 '24 at 01:36
  • 1
    Fair enough. I answer the question as written, and in any case, it is still a very useful question (+1) because this is an attitude I have encountered many times (so answer hopefully useful). – Ben Feb 01 '24 at 01:40
  • This answer reminds of one you did a while back that answered in the style of some post-modern philosophy. – Galen Feb 01 '24 at 01:44
  • 2
    "this is an attitude I have encountered many times (so answer hopefully useful)" Totally fair! And I'll leave my lazy writing there, with my cart firmly ahead of my horse, so your answer remains valid. Maybe passersby will learn the lesson. – kdbanman Feb 01 '24 at 01:56
  • 1
    @kdbanman: Thanks for being such a good sport --- I think your question contributes something valuable to this site. – Ben Feb 01 '24 at 02:02
  • 1
    @Galen: Here it is. – Ben Feb 01 '24 at 02:04
  • 2
    Given the formulation of NHST so often pounded into students' heads - "NEVER say we accept the null hypothesis that the means are equal, we always either reject the null or, if the test is not significant, fail to reject it" - my immediate instinct was to read "want" in the less literal sense of "actually no, I really am interested in whether there's evidence to accept the null, thank you very much". Bit of a shame the question title didn't add a parenthetic "What do I do when I want to accept (or not) the usual null hypothesis?" – Silverfish Feb 01 '24 at 02:04
  • 1
    I think "want" in the sense of "it'd be more interesting to be able to talk about whether evidence supports the null, than whether it rejects it" is a fair enough starting point for a question - not unproblematic (is anything about the whole NHST framework unproblematic?!) but the expression of wanting, suggesting a preconceived or desired result, is just one small part of the problem. (And arguably part of the problem with the conventional "can we reject H0" approach too.) I also suspect the "NEVER accept the null" maxim is part of why people are surprised when they discover TOST is a thing! – Silverfish Feb 01 '24 at 02:09
  • 1
    @kdbanman: Incidentally, if you had an alternative question in mind (absent the lazy writing) that is a bit more subtle, feel free to post a new question and I will be happy to answer it with a less snarky answer. ;) – Ben Feb 01 '24 at 02:11
  • 1
    I think it'd be perfectly valid to post both a snarky answer and a serious answer to this question. The core intent is perfectly clear here. The point about sloppy language is well-made and worth making, but even if you tighten up the language, this notion of "wanting" is implicit, lurking not far below the surface, in almost all practical applications of the NHST. Even in the conventional "is there sufficient evidence to reject H0" approach, like I said above: how many times have you read a paper and sensed an authorial fist pump every time $p<0.05$ or they got an extra * in their table? – Silverfish Feb 01 '24 at 02:21
  • @Silverfish re your comment about lurking in most applications of NHST, thank god it doesn't happen with Bayesian applications.! – Graham Bornholt Feb 01 '24 at 03:04
  • @GrahamBornholt Sarcasm? – Galen Feb 01 '24 at 04:30
  • @Galen Yes, using humour to give a little balance. – Graham Bornholt Feb 01 '24 at 07:35
  • 2
    @Silverfish your phrasing "it'd be more interesting to be able to talk about whether evidence supports the null, than whether it rejects it" sums up the essence of my question! – kdbanman Feb 01 '24 at 07:40
  • 1
    I’ll consider asking another, more carefully crafted question about hypothesis tests for equivalence of variance or other properties. There are other areas I’d like to know how to reject (or fail to reject) the alternate hypotheses! – kdbanman Feb 01 '24 at 07:49
  • @Ben the more reasonable question is up here! If you still have time for an answer or a clue, it'd be much appreciated. – kdbanman Feb 05 '24 at 00:51