Narrow confidence interval -- higher accuracy?

Question

I have two questions about confidence intervals:

Apparently a narrow confidence interval implies that there is a smaller chance of obtaining an observation within that interval, therefore, our accuracy is higher.

Also a 95% confidence interval is narrower than a 99% confidence interval which is wider.

The 99% confidence interval is more accurate than the 95%.

Can someone give a simple explanation that could help me understand this difference between accuracy and narrowness?

Answer 1

The 95% is not numerically attached to how confident you are that you've covered the true effect in your experiment. Perhaps recognizing that 95% is attached to the procedure that produced the interval, and not the interval itself, would help. Part of the procedure is that you decide that the interval contains the true value. You'll be right if you do that consistently 95% of the time. But you really don't know how likely it is for your particular experiment without more information.

Q1: Your first query conflates two things and misuses a term. A narrower confidence interval may be more precise but it's accuracy is fixed by the procedure backing it, be it 89%, 95%, etc. As long as that procedure was correctly designed, the true value will be contained in the interval at the prescribed rate. (see @Michael R Chernick's answer for a discussion on coverage and a different sense of accuracy)

Whether an observation falls in a CI isn't something to consider. A confidence interval is about estimating the mean. If you had an extraordinary large sample size and could estimate the mean very well then the chances of an observation being in the CI would be miniscule.

Nevertheless, your question does raise some points and it's important to think about why a CI is narrow. Just because it's narrow doesn't mean you're less likely to encounter an observation that falls within that CI. Keep in mind, a narrow CI can be achieved in one of three ways. The most common way researchers make the CI narrow is by increasing the sample size. Another way they can be narrow is because the experimental method or nature of the data yields very low variance. For example, the confidence interval around the boiling point of water at sea level is small, regardless of the sample size. Finally, it could be narrow because your sample is unrepresentative. In that case, you are actually more likely to have one of the 5% of intervals that do not contain the true value. It's a bit of a paradox regarding CI width that the ones in that 5% of misses tend to be narrow. It's something you should check by knowing the literature and how variable this data typically is.

Q2: A 99% confidence interval is wider than a 95%, all else being equal. Therefore, it's more likely that it will contain the true value. See the distinction above between precise and accurate. If I make a confidence interval narrower with lower variability and higher sample size it becomes more precise because the values cover a smaller range. If I increase the coverage by using a 99% calculation it becomes more accurate because the true value is more likely to be within the range.

Answer 2

For a given dataset, increasing the confidence level of a confidence interval will only result in larger intervals (or at least not smaller). That's not about accuracy or precision but rather about how much risk you're willing to take about missing the true value.

If you're comparing confidence intervals for the same sort of parameter from multiple data sets and one is smaller than the other, you could say that the smaller one is more precise. I prefer to talk about precision rather than accuracy in this situation (see this relevant Wikipedia article).

Answer 3

First of all, a CI for a given confidence percentage (e.g.95%) means, for all practical purposes (though technically it is not correct) that you are confident that the true value is in the interval.

If this is interval is "narrow" (note that this can only be regarded in a relative fashion, so, for comparison with what follows, say it is 1 unit wide), it means that there is not much room to play: whichever value you pick in that interval is going to be close to the true value (because the interval is narrow), and you are quite certain of that (95%).

Compare this to a relatively wide 95% CI (to match the example before, say it is 100 units wide): here, you are still 95% certain that the true value will be within this interval, yet that doesn't tell you very much, since there are relatively many values in the interval (about a factor 100 as opposed to 1 - and I ask, again, of purists to ignore the simplification).

Typically, you are going to need a bigger interval when you want to be 99% certain that the true value is in it, than when you only need to be 95% certain (note: this may not be true if the intervals are not nested), so indeed, the more confidence you need, the broader the interval you will need to pick.

On the other hand, you are more certain with the higher confidence interval. So, If I give you 2 intervals of the same width, and I say one is a 95% CI and the other is a 99% CI, I hope you will prefer the 99% one. In this sense, 99% CIs are more accurate: you have less doubt that you will have missed the truth.

Answer 4

I am adding to some good answers here that I gave upvotes to. I think there is a little more that should be said to completely clear up the conclusion. I like the terms accurate and correct as Efron defines them. I gave a lengthy discussion on this very recently on a different question. The moderator whuber really liked that answer. I will not go to the same length to repeat that here. However, to Efron accuracy relates to the confidence level and correctness of the width or tightness of the interval. But you can't talk about tightness without considering accuracy first. Some confidence intervals are exact; those are accurate because they have the actual coverage that they advertise. A 95% confidence interval can also be approximate because it uses an asymptotic distribution. Approximate intervals based on asymptotics are for a finite sample size $n$ not going to have the advertised coverage, which is the coverage you would get if the asymptotic distribution were the exact distribution.

So an approximate interval could undercover (i.e. advertise 95% when its actual coverage is only 91%) or in the rare but less serious case overcover (i.e. advertised coverage is 95% but actual in 98%). In the former case, we worry about how close the actual coverage is to the advertised coverage). A measure of closeness is the order of accuracy which could be say $1/\sqrt{n}$ or $1/n$. If the actual confidence level is close, we call it accurate. Accuracy is important with bootstrap confidence intervals which are never exact but some variants are more accurate than others.

This definition of accuracy may be different to the one the OP is referring to but it should be clear now what Efron's definition is and why it is important to be accurate. Now if you have two methods that are exact, we can prefer one over the other if for any confidence level it has the smaller expected width. A confidence interval that is best in this sense (sometimes called the shortest) would be the one to choose. But this required exactness. If the confidence level is only approximate we could be comparing apples and oranges. One could be narrower than another only because it is less accurate and hence has a lower actual coverage than its advertised coverage.

If two confidence intervals are both very accurate or one is exact and the other very accurate comparing expected width may be okay because at least now we are looking at just two varieties of apples.

Answer 5

For a fixed dataset there is a trade-off between accuracy and confidence

Remember that when you have a fixed dataset, this give you a limited amount of information, and so you would expect that there is "no free lunch". That is, for a fixed dataset you must get a trade-off between accuracy and confidence --- if you want more accuracy then you have to settle for less confidence, and if you want more confidence you have to settle for less accuracy.

This information constraint is the reason you observe the stated relationship between the width of the confidence interval (accuracy) and the confidence level (confidence). If you increase the confidence level then you will get a wider confidence interval, making it less accurate. Conversely, if you decrease the confidence level then you will get a narrower confidence interval, making it more accurate. This is a general property of sensible confidence interval procedures that maximise use of the available information in the dataset.

It is possible to examine this relationship mathematically for a given confidence interval procedure. Usually we have a procedure that leads to a $1-\alpha$ level confidence interval of the form:

$$\text{CI}(n,\alpha) \equiv [L(n,\alpha), U(n,\alpha)],$$

where $n$ is any a positive real sample size.$^\dagger$ Letting $W(n,\alpha) \equiv U(n,\alpha) - L(n,\alpha)$ denote the width of the confidence interval, this function will typically have the monotonicity properties:

$$\frac{\partial W}{\partial \alpha}(n,\alpha) < 0 \quad \quad \quad \frac{\partial W}{\partial n}(n,\alpha) < 0.$$

The first monotonicity property means that when you decrease the confidence level (by increasing $\alpha$) you get a more accurate (narrower) confidence interval and vice versa. The second monotonicity property means that when you get more sample data, you get a more accurate (narrower) confidence interval at each confidence level.

If you look at the relevant confidence interval formulae for a particular procedure, you will typically be able to confirm these monotonicity properties. (In the rare case that one of these monotonicity properties does not hold, it would raise questions about the rationality of the confidence interval procedure.) If you would like to see an example of analysis of monotonicity properties of a confidence interval procedure, see e.g., O'Neill (2021).

---

$^\dagger$ Of course, in practice the sample size is a positive integer but it is usually the case that the confidence interval formula can be extended to any positive real sample size in a natural way and then it can be examined using standard calculus methods.