6

It seams like confidence intervals resemble critical regions in hypothesis testing .. are there cases when one cannot use these intervals in testing?

In critical regions we use the mean and see where it lands while in confidence intervals we use the mean to construct the boundaries.

1 Answers1

4

Yes, there are situations where tests cannot be executed through confidence sets. Generally speaking, this happens in inferential problems where interest is not on a (sub)set of parameters but on the model itself. Goodness-of-fit tests are one such example.

For instance, when in a linear regression model we want to test if the residuals are normally distributed. The typical procedure for the test is:

  • compute the observed statistics and the associate $p$-value, then compare the $p$-value with the type I error $\alpha$, or
  • compute the observed statistic and compare it with the threshold value obtained from the (typically) asymptotic distribution of the chosen test.

As far as I know, there is no confidence set procedure that can tell us if the residuals are normal or not.

UPDATE @kjetil b halvorsen points to this confidence set procedure obtained by inverting the Kolmogorov-Smirnov test statistic.

utobi
  • 11,726
  • 3
    I believe there are at least some potential intervals that correspond to some goodness of fit hypothesis tests. One example is that you can place an interval around an ecdf that corresponds to a Lilliefors test of normality. I think you can also get confidence-interval based test equivalents with Shapiro-Francia and Ryan-Joiner tests. – Glen_b Oct 02 '22 at 10:59
  • 2
    thanks @Glen_b, I have never seen these procedures. Do you have any reference? I'd be happy to include them in my answer. – utobi Oct 02 '22 at 11:52
  • 2
    See https://stats.stackexchange.com/questions/298290/plotting-non-parametric-ecdef-confidence-envelopes-for-comparison/298495#298495 for some refs – kjetil b halvorsen Oct 02 '22 at 13:17
  • 1
    @utobi $1.$ https://en.wikipedia.org/wiki/Lilliefors_test $2.$ https://en.wikipedia.org/wiki/Shapiro%E2%80%93Francia_test $3.$ (a) Ryan T A, Joiner B L (1976) Normal Probability Plots and Tests for Normality," Technical Report, Statistics Department, The Pennsylvania State University . . .

    (b) Stephen W. Looney & Thomas R. Gulledge, Jr. “Use of the Correlation Coefficient with Normal Probability Plots”, The American Statistician, Vol. 39, No. 1 (Feb. 1985), pp. 75-79.

     – Glen_b Oct 03 '22 at 06:25