3

I'm using a service (Optimizely, as it happens), which show a confidence interval but doesn't explain what it is.

I'm assuming it is a Binomial Confidence Interval, but I have no idea what the % certainty of the range is, or the detailed meaning. Searching the Internet doesn't help - there is nobody with a clear explanation of what the default for this kind of thing is.

For example, for 97 success out of 939 trials, Optimizely says "10.33% (±1.95)".

What does that ±1.95 likely mean, exactly?

And yes, I've already complained to them that they should document it. But clearly, there is some default that statisticians habitually use. What is it?

frabcus
  • 133
  • Talking about this a bit more in the office here at ScraperWiki, it seems in some disciplines there is a different assumption. e.g. Physics uses 1 sigma (68%) only quite often. Would love good in answer to this in an answer too! – frabcus Jun 24 '14 at 14:44

3 Answers3

3

Exactly? You want an exact answer to a statistical question? And you're willing to pay a service to provide it? Did you formerly work for AIG?

Seriously, we will need to guess here. My guess is that you are being offered an estimate of the 95% confidence interval using a Normal approximation around the point estimate of the binomial proportion for your data. My guess is informed by the fact that I have an open R console session:

> binom.test(97, 939,  10.33/100)

    Exact binomial test

data:  97 and 939
number of successes = 97, number of trials = 939, p-value = 1
alternative hypothesis: true probability of success is not equal to 0.1033
95 percent confidence interval:
 0.08457063 0.12456106
sample estimates:
probability of success 
             0.1033014 

> abs(0.08457063 - 0.12456106)
[1] 0.03999043
> abs(0.08457063 - 0.12456106)/2
[1] 0.01999522

There's an "exact" answer, but exact has a different meaning than the word you are probably asking. Furthermore, you will notice that one half the span of the 95% CI 1.99995% is not the same as your Optimizely answer of 1.95%. The exact CI's are not symmetric when the binomial proportion is far away for 0.5.

DWin
  • 7,726
2

Following DWin's answer, I used R to find some more binomial confidence intervals:

> library(binom)
> binom.confint(97, 939)
          method  x   n      mean      lower     upper
1  agresti-coull 97 939 0.1033014 0.08535696 0.1244784
2     asymptotic 97 939 0.1033014 0.08383472 0.1227681
3          bayes 97 939 0.1037234 0.08452740 0.1233938
4        cloglog 97 939 0.1033014 0.08487358 0.1237776
5          exact 97 939 0.1033014 0.08457063 0.1245611
6          logit 97 939 0.1033014 0.08539352 0.1244537
7         probit 97 939 0.1033014 0.08514200 0.1241205
8        profile 97 939 0.1033014 0.08492041 0.1238435
9            lrt 97 939 0.1033014 0.08491887 0.1238435
10     prop.test 97 939 0.1033014 0.08493732 0.1249860
11        wilson 97 939 0.1033014 0.08542358 0.1244118

"10.33% (±1.95)" looks like the asymptotic 95% confidence interval.

user20637
  • 716
  • 5
  • 11
1

The Optimizely documentation says:

What these numbers mean is that at the time the experiment was running, if you were to take a bunch of random samples, 95% of the time, the conversion rate would fall in the range listed. For example, if the conversion rate is 6.0% and the confidence interval is 0.11%, then 95% of the tests will have a conversion rate between 5.89% and 6.11%.

More generally, Wikipedia suggests this is common practice:

In applied practice, confidence intervals are typically stated at the 95% confidence level. However, when presented graphically, confidence intervals can be shown at several confidence levels, for example 50%, 95% and 99%.

And discussion with friends suggests it's field specific.

user48921
  • 111