12

In the wikipedia article on Credible Interval, it says:

For the case of a single parameter and data that can be summarised in a single sufficient statistic, it can be shown that the credible interval and the confidence interval will coincide if the unknown parameter is a location parameter (i.e. the forward probability function has the form Pr(x | μ) = f(x − μ) ), with a prior that is a uniform flat distribution;[5] and also if the unknown parameter is a scale parameter (i.e. the forward probability function has the form Pr(x | s) = f(x / s) ), with a Jeffreys' prior [5] — the latter following because taking the logarithm of such a scale parameter turns it into a location parameter with a uniform distribution. But these are distinctly special (albeit important) cases; in general no such equivalence can be made."

Could people give specific examples of this? When does the 95% CI actually correspond to "95% chance", thus "violating" the general definition of CI?

Wayne
  • 21,174

1 Answers1

16

normal distribution:

Take a normal distribution with known variance. We can take this variance to be 1 without losing generality (by simply dividing each observation by the square root of the variance). This has sampling distribution:

$$p(X_{1}...X_{N}|\mu)=\left(2\pi\right)^{-\frac{N}{2}}\exp\left(-\frac{1}{2}\sum_{i=1}^{N}(X_{i}-\mu)^{2}\right)=A\exp\left(-\frac{N}{2}(\overline{X}-\mu)^{2}\right)$$

Where $A$ is a constant which depends only on the data. This shows that the sample mean is a sufficient statistic for the population mean. If we use a uniform prior, then the posterior distribution for $\mu$ will be:

$$(\mu|X_{1}...X_{N})\sim Normal\left(\overline{X},\frac{1}{N}\right)\implies \left(\sqrt{N}(\mu-\overline{X})|X_{1}...X_{N}\right)\sim Normal(0,1)$$

So a $1-\alpha$ credible interval will be of the form:

$$\left(\overline{X}+\frac{1}{\sqrt{N}}L_{\alpha},\overline{X}+\frac{1}{\sqrt{N}}U_{\alpha}\right)$$

Where $L_{\alpha}$ and $U_{\alpha}$ are chosen such that a standard normal random variable $Z$ satisfies:

$$Pr\left(L_{\alpha}<Z<U_{\alpha}\right)=1-\alpha$$

Now we can start from this "pivotal quantity" for constructing a confidence interval. The sampling distribution of $\sqrt{N}(\mu-\overline{X})$ for fixed $\mu$ is a standard normal distribution, so we can substitute this into the above probability:

$$Pr\left(L_{\alpha}<\sqrt{N}(\mu-\overline{X})<U_{\alpha}\right)=1-\alpha$$

Then re-arrange to solve for $\mu$, and the confidence interval will be the same as the credible interval.

Scale parameters:

For scale parameters, the pdfs have the form $p(X_{i}|s)=\frac{1}{s}f\left(\frac{X_{i}}{s}\right)$. We can take the $(X_{i}|s)\sim Uniform(0,s)$, which corresponds to $f(t)=1$. The joint sampling distribution is:

$$p(X_{1}...X_{N}|s)=s^{-N}\;\;\;\;\;\;\;0<X_{1}...X_{N}<s$$

From which we find the sufficient statistic to be equal to $X_{max}$ (the maximum of the observations). We now find its sampling distribution:

$$Pr(X_{max}<y|s)=Pr(X_{1}<y,X_{2}<y...X_{N}<y|s)=\left(\frac{y}{s}\right)^{N}$$

Now we can make this independent of the parameter by taking $y=qs$. This means our "pivotal quantity" is given by $Q=s^{-1}X_{max}$ with $Pr(Q<q)=q^{N}$ which is the $beta(N,1)$ distribution. So, we can choose $L_{\alpha},U_{\alpha}$ using the beta quantiles such that:

$$Pr(L_{\alpha}<Q<U_{\alpha})=1-\alpha=U_{\alpha}^{N}-L_{\alpha}^{N}$$

And we substitute the pivotal quantity:

$$Pr(L_{\alpha}<s^{-1}X_{max}<U_{\alpha})=1-\alpha=Pr(X_{max}L_{\alpha}^{-1}>s>X_{max}U_{\alpha}^{-1})$$

And there is our confidence interval. For the Bayesian solution with jeffreys prior we have:

$$p(s|X_{1}...X_{N})=\frac{s^{-N-1}}{\int_{X_{max}}^{\infty}r^{-N-1}dr}=N (X_{max})^{N}s^{-N-1}$$ $$\implies Pr(s>t|X_{1}...X_{N})=N (X_{max})^{N}\int_{t}^{\infty}s^{-N-1}ds=\left(\frac{X_{max}}{t}\right)^{N}$$

We now plug in the confidence interval, and calculate its credibility

$$Pr(X_{max}L_{\alpha}^{-1}>s>X_{max}U_{\alpha}^{-1}|X_{1}...X_{N})=\left(\frac{X_{max}}{X_{max}U_{\alpha}^{-1}}\right)^{N}-\left(\frac{X_{max}}{X_{max}L_{\alpha}^{-1}}\right)^{N}$$

$$=U_{\alpha}^{N}-L_{\alpha}^{N}=Pr(L_{\alpha}<Q<U_{\alpha})$$

And presto, we have $1-\alpha$ credibility and coverage.

Jorge Leitao
  • 1,359
  • A masterpiece, thanks! I was hoping that there might be an answer like, "when calculating the mean of a sample from a Normal distribution, the 95% CI actually is also the 95% Credible Interval" or something simple like that. (Just making up this supposed answer, I have no clue as to specific examples.) – Wayne Jul 02 '11 at 18:07
  • I believe that a frequentist 95% prediction/tolerance interval corresponds to a Bayesian prediction interval with OLS regression and normal errors. It appears so when I compare predict.lm's answer with a simulated answer, anyhow. Is that true? – Wayne Jul 07 '11 at 18:16
  • For $Y=\alpha+\beta X$, If you use a uniform prior for $\alpha,\beta$ and jeffreys prior for $\sigma$, then you have equivalence. – probabilityislogic Jul 08 '11 at 00:17
  • Great, thanks! I've been trying to explain a CI for a regression I've done in terms of a Confidence Interval, and it simply doesn't connect with a layman audience, which expects a Credible Interval. Makes life much easier for me... though perhaps it's bad for the overall statistical world, as it will reinforce the layman's misunderstanding of CI's. – Wayne Jul 08 '11 at 14:21
  • @Wayne - the situation is a little more general than just location scale families. Usually a CI will be equivalent to credible interval, if it is based on a "sufficient statistic" (as these two were) where this exists. If no sufficient statistic, then CI needs to condition on what are called "ancillary statistics" to have credible interval interpretation. – probabilityislogic Jul 09 '11 at 02:04
  • Thanks so much for your insight. I hope others find this useful, and continue to contribute. My main curiosity is that "Confidence Interval does not mean 95% chance" is oft-repeated, and it is the layman who misinterprets the CI as a credibile interval, but in how many common instances is the layman's misinterpretation in fact correct? Back to your latest comment: is this another case where that "normal curve" pops up again (i.e. do normality assumptions imply sufficient statistics)? – Wayne Jul 11 '11 at 17:10
  • @Wayne Indeed, it's very relevant to know exactly under what circumstances (e.g. under what type of uninformative prior) a credible interval corresponds exactly to a frequentist confidence interval, and the common misinterpretation of "a 95% confidence interval encompassing the true value with 95% chance" would in fact be correct. Also relevant here: https://arxiv.org/pdf/1211.3343.pdf – Tom Wenseleers Apr 09 '21 at 16:48