4

Say I have three sets of observations, $A$, $B$, and $C$ (say, the heights of Australians, Americans and Brits). I assume that they are all normally distributed, independent etc.

I perform two-sided t-tests and find that I cannot reject $H_0: \bar{x}_A - \bar{x}_B = 0$ nor $H_0: \bar{x}_B - \bar{x}_C = 0$. I know that a t-test does not "prove" the null hypothesis, but should I expect transitivity - should a third t-test also fail to reject $H_0: \bar{x}_A - \bar{x}_C = 0$?

Furthermore, if it seems that $A$, $B$ and $C$ are being drawn from the same population (seemingly the same mean, assumedly the same distribution), should I expect that the mean of any given sample will not be statistically significantly different to the mean of the union of all three? Should a t-test fail to reject any of the hypotheses $\bar{x}_{A \cup B \cup C} - \bar{x}_{A|B|C} = 0$?

Michael K
  • 43
  • Welcome to the site, Michael! I have a hard time believing this is the first time this question has come up on this site, but an even harder time finding the first one. Some similar ones:

    https://stats.stackexchange.com/questions/83030/can-anova-be-significant-when-none-of-the-pairwise-t-tests-is

    https://stats.stackexchange.com/questions/3549/why-is-it-possible-to-get-significant-f-statistic-p-001-but-non-significant-r

     – eric_kernfeld Jan 15 '19 at 22:30
  • 1
    Not necessarily. the estimates could show that the mean of A is statistically far enough away from the mean for C to reject the null hypothesis. – Michael R. Chernick Jan 15 '19 at 22:32
  • 5
    If true, this would lead to a modern Zeno paradox in which you would demonstrate the impossibility of distinguishing between any means because of the lack of significance in distinguishing between any neighboring pairs of a closely spaced set of means between the original two means. For instance, if this transitivity were true and you were to take a digital photograph of extremely high resolution, from the similarity of all pixels in every small neighborhood you would conclude that no features can be distinguished anywhere in the picture! – whuber Jan 15 '19 at 22:51

1 Answers1

4

It's not clear in what sense you are using the word "expect". You should roughly expect these things to go down the way you describe but it is not granted. My answer will assume that you used "expect" in a strict sense (i.e. should always be the case). Then:

The answer to the first question is no.

The reason is that there might be a bigger gap between $a$ and $c$ then there were between $a-b$ and $b-c$. And here is a small example:

a = 1, 2, 3
b = 3, 4, 5
c = 5, 6, 7

with the above example at $\alpha = 0.05$ both $a-b$ and $b-c$ comparisons would not be significantly different from 0 but $a-c$ comparison would be.

The answer to the second question is also in the negative. The reason for that one is the random element in the sampling. You could (with very small probability) obtain these numbers when sampling from the same distribution:

a = 0, 0, 0
b = 1, 1, 1
c = 1, 1, 1

And $abc$ - $a$ would be significantly different from 0. So you have no guarantee.

Karolis Koncevičius
  • 5,041