1

Assume that we have two scalar quantities $X$ and $Y$, and a confidence interval oracle $CI$ for each of them, so that $CI_X(\alpha)$ and $CI_Y(\alpha)$ will produce level-$\alpha$ confidence intervals for $X$ and $Y$ respectively. Assume that we know nothing about the procedure by which the confidence intervals are generated, that they are not necessarily symmetric, but that they are exact.

What would be the most reasonable way to test a hypothesis of the form $X=Y$? This question contains work on t tests specifically, and seems to have quite strong results in the answer by @whuber, but under assumptions on things such as the underlying variance. What is there for a more general, non-symmetric, non-normal case?

If two confidence intervals at level $\alpha$ do not overlap at all, it seems intuitively reasonable to at least say that we can reject the null hypothesis $X=Y$ at a $p < \alpha$ level. Does this have a theoretical justification, and is there anything stronger?

Marc Vaisband
  • 301
  • 2
    Because you appear to use the same terms and symbols both for data and for parameters, could you clarify what is going on? For instance, if "$X$" and "$Y$" refer to parameters, then what are the data the confidence intervals are based on? One concern is that if these intervals are based on a common dataset, then likely they are not independent. – whuber Feb 26 '24 at 16:34
  • 1
    @whuber The datasets are separate - in fact, that's the aim. Let's say, for example, that we do parameter estimation with uncertainty on two data groups, and wish to compare the same parameter between the two fits. (And let's assume that we can't do a joint model or similar where we'd get a confidence interval on the difference directly) – Marc Vaisband Feb 26 '24 at 21:01
  • 1
    That makes the second part of your question easy to answer: because under the null hypothesis each interval has a $1-\alpha$ chance of covering the (common) parameter value, they will overlap with at least a $(1-\alpha)^2$ probability. Thus, lack of overlap occurs with a chance of at most $1-(1-\alpha)^2.$ (With symmetric intervals this calculation can be improved.) – whuber Feb 26 '24 at 22:27

0 Answers0