3

I've come across the phrase "honest confidence interval" but I have never seen a definition. What is it?

amrods
  • 249
  • 1
    I haven't heard of it before but a quick search leads to: https://www.jstor.org/stable/2241707 – Sextus Empiricus Oct 08 '22 at 13:00
  • 1
    In what context did you read it? I can speculate (I think correctly) that it means a confidence interval at the $(1-\alpha)\times 100%$-level will contain the true parameter in the claimed percentage of replications, but I’ve never heard this term before. – Dave Oct 08 '22 at 13:02
  • How is that different than the usual definition? – amrods Oct 08 '22 at 13:16
  • 1
    I might claim a confidence of $95%$, but who is to say that the procedure really would cover the true parameter in $95%$ of replications? – Dave Oct 08 '22 at 13:21
  • I don't follow. Doesn't a regular CI cover the parameter in 95% of the cases? For example, the CI for the mean assuming normality of observations: $\Pr(\mu \in [\bar{x} \pm t s/\sqrt{n}]) = 95%$. – amrods Oct 08 '22 at 13:26
  • Then that would be an honest confidence interval. However, if you replace $s$ with some other estimate of the standard deviation and $\bar x$ with some other estimate of the mean, that procedure might not be. – Dave Oct 08 '22 at 13:30
  • Ah I see. I would argue that that would not be a CI in the first place. – amrods Oct 08 '22 at 13:36
  • 1
    A commonly occuring example of "some other estimate" in that case might be "the mean and standard deviation after removing seeming outliers" or "the mean and variance after looking at the data to see if some transformation might be needed". The usual pivotal quantity used to derive the interval is no longer distributed as t. – Glen_b Oct 09 '22 at 10:02
  • 1
    @SextusEmpiricus the project euclid link offers a free-access pdf for IMS publications (and others; it hosts over 100 journals); the paper is available here: https://projecteuclid.org/journals/annals-of-statistics/volume-17/issue-3/Honest-Confidence-Regions-for-Nonparametric-Regression/10.1214/aos/1176347253.full – Glen_b Oct 09 '22 at 10:12

1 Answers1

7

An example of an article that defines it is "Honest Confidence Regions for Nonparametric Regression" by Ker-Chau Li (1989).

Here the word "honest" refers to the requirement that the minimum coverage probability over a rich class of (nonparametric) regression functions should be no less than the nominal confidence level

Confidence intervals can 'fail' to be honest when they are not exact computations, and are instead estimates.

An example of how it can go 'wrong' are the different methods to compute a confidence intervals for a binomial proportion. The Clopper-Pearson is always covering at least the given percentage, no matter what the true proportion parameter is. Other intervals, like Wald interval or Jeffreys interval don't (at least not when you condition on the parameter).

The image below gives an example where the coverage probability is given as function of the true parameter for the case of a sample from a binomial distribution with n=100 (From this question Revisiting the Rule of Three)

example of coverage probability for different intervals

In the case of the Wald interval the 'mistakes' are introduced because it is an approximation. In the case of Jeffreys' interval the 'mistake' is because the interval is actually a credible interval and doesn't even try to be a confidence interval by design.

Sextus Empiricus
  • 77,915
  • 2
    That's really interesting; I always thought that achieving a certain coverage rate was actually the definition of a confidence interval tout court. – John Madden Oct 08 '22 at 14:30
  • 3
    @JohnMadden that definition is still correct. It is only that what we often use as confidence intervals are in fact approximations of confidence intervals. – Sextus Empiricus Oct 08 '22 at 15:06
  • I was in that same bus. – amrods Oct 08 '22 at 21:17
  • The extract in the answer appears to use the word 'honest' to signify a type of robustness of CI coverage across models. I think we are fortunate that the use of the word honest in this way never caught on. – Graham Bornholt Oct 08 '22 at 22:38
  • Also, the Jeffrey's credible region is not a confidence interval so it is a misnomer to call it a confidence region. While the coverage rates of Bayesian credible intervals is of interest, they should not be renamed as confidence intervals. – Graham Bornholt Oct 08 '22 at 22:48
  • @GrahamBornholt I have edited the answer to make the situation with Jeffreys' interval more clear. – Sextus Empiricus Oct 09 '22 at 00:42
  • 1
    @SextusEmpiricus That works. – Graham Bornholt Oct 09 '22 at 01:37