1

According to to G. Cumming and S. Finch in their paper "Inference by Eye" the 95% confidence intervals of two independent means can overlap and the difference between the two means can still be significantly different from 0. They found that at approximately 58% overlap the p-value of the t-test changes from significant to not significant at 0.05.

Now to my question.

Does this apply for all types of confidence intervals? Im mostly interested if this holds for two independent proportions. If yes, why is that? And if no, why is that?

User1865345
  • 8,202
Jam.Wil
  • 77
  • 1
    Is the confidence interval approximated using the normal distribution? – user2974951 Nov 22 '23 at 13:08
  • For the two proportions? Yes the confidence intervals is approximated using the normal distribution. In my case n1 and n2 are both > 10 000 and p1 and p2 are both around 0,1. – Jam.Wil Nov 22 '23 at 13:14
  • 2
    https://stats.stackexchange.com/questions/18215 answers your question generally and offers a more specific, quantitative solution than the paper you reference. – whuber Nov 22 '23 at 14:48

1 Answers1

1

I hesitate to say it applies to all types of confidence intervals, there could be some odd cases. But it generally applies (although it can get tricky).

Why?

Well, a p-value of x (say, 0.05) would match a 1-x (0.95) confidence interval vs. a constant. For the two means case, it would be that the CI for one proportion doesn't cover the other propotion (not its CI, but the value itself).

It gets tricky for more complex notions. E.g. if we have p means, then which are overlapping which? And what the "other" CI would be is not at all clear for, say, an F test. But the idea is generally sound. It certainly applies to, say, a t-test of means rather than proportions.

Peter Flom
  • 119,535
  • 36
  • 175
  • 383