0

Wondering if Fieller's theorem is used for the 'ratio' value in the compare argument in statsmodels.stats.proportion.confint_proportions_2indep method from the statsmodel module. If not, what is it? I'm trying to display the confidence interval of the % difference between p1-p2

Also, how can I interpret the numbers. I got the following 95% conf intervals with ratio: (1.05, 1.31).

EDIT: When computing for the risk ratio, the 95%CI is :(1.05, 1.31) What does it mean? Is that that the click through rate for the control group (metric being analyzed) is between 5% and 31% more likely to be clicked than the treatment group ?

Data: P1 = 30.97% n=50000, P2 = 26.34%, n=50050 Delta = -14.9% How can i get the CI for the Delta?

kjetil b halvorsen
  • 77,844
Roger Steinberg
  • 211
  • What would "Filler's Theorem" be? I am unaware of any commonly known result that goes by that name. Fieller's Theorem concerns a ratio of means, not a difference, and so doesn't seem to be what you intend. – whuber May 23 '22 at 17:35
  • @whuber Yes I meant Fieller's. If two proportions difference is say 10% , what would help me calculate its confidence interval ? Fieller's theorem doesn't do that ? What about the function in my question using 'ratio'? – Roger Steinberg May 23 '22 at 17:38
  • 1
    You appear to be asking two different questions. Do you need a confidence interval for the difference or for their ratio? Concerning your bold question, that's a matter of what a confidence interval means: you can find good information at https://stats.stackexchange.com/questions/26450. – whuber May 23 '22 at 19:02
  • PLease clairify your question, as asked for! In the meantime, have a look at https://stats.stackexchange.com/questions/21298/confidence-interval-around-the-ratio-of-two-proportions, https://stats.stackexchange.com/questions/115885/confidence-interval-for-the-ratio-of-two-related-proportions-glass-ceiling-inde, https://stats.stackexchange.com/questions/544286/in-r-trying-to-find-the-confidence-interval-for-the-difference-in-proportions and search this site! – kjetil b halvorsen May 23 '22 at 19:16
  • @whuber for the difference % – Roger Steinberg May 23 '22 at 19:20
  • @kjetilbhalvorsen finding the confidence interval for the difference between both proportions is simple. what I cant seem to figure out is how to determine the confidence interval to the difference in %. So if p1 = 0.20 and p2=0.14 that's a 30% difference. And what I need is the 95% CI that goes with that statistic. – Roger Steinberg May 23 '22 at 19:27
  • Please add that clarification as an edit to the post! We want posts to be self-contained, and comments are easily overlooked. Then also clarify what data you have, and assumption we can use. Is this binomial data? Maybe add some example data (mock-up data, if you want) to your post. My answer at https://stats.stackexchange.com/questions/289008/confidence-interval-on-binomial-effect-size/406746#406746 might be a similar case, but that uses R ... – kjetil b halvorsen May 23 '22 at 19:33
  • @kjetilbhalvorsen just did – Roger Steinberg May 23 '22 at 19:38
  • OK, seen. But you have only given %, for a confidence interval we need actual counts. With counts the method in my last link can be adapte, and that do not need Fiellers theorem! – kjetil b halvorsen May 23 '22 at 19:40
  • The difference between $p_1=0.20=20%$ and $p_2 = 0.14=14%$ is, by definition, $20%-14% = 6%.$ Their ratio is $0.20/0.14=1.42.$ Regardless of which one of those you need--and they have different meanings--the subject is still fraught: many of the traditional procedures generally work poorly and choosing a suitable one depends, inter alia, on (a) the amounts of data for each proportion; (b) whether the data are matched or independent; (c) whether you need to guarantee minimum coverage; and (d) how extreme (close to 0 or 1) the $p_i$ might be. – whuber May 23 '22 at 19:45
  • @kjetilbhalvorsen added the size – Roger Steinberg May 23 '22 at 19:54
  • @whuber , im still very new to ab testing and im trying to get my head around confidence intervals in general. mostly, i've been dispalying results in terms of the absolute difference between both proportions ('diff' in the function above). I wanted to know simply if the CI level can be used in conjunction with the delta % of both proportions. Simply put: Control performed better by X% and its 95% CI level is Z%-Y%. I thought ratio would provide somewhat of similar response but based on your answers it doesn't. – Roger Steinberg May 23 '22 at 19:59
  • 1
    Re the edit and recent comments: In AB testing usually the datasets are so large you can use Normal-theory approximations. See https://stats.stackexchange.com/questions/388294 for instance. – whuber May 23 '22 at 21:00
  • @whuber right thats the absolute difference. Do you know what the ratio would mean or do you know if there is a way to get CI for % difference instead? – Roger Steinberg May 23 '22 at 21:39
  • 1
    For such large datasets--as I wrote--Normal approximations will be fine. One source of software for these calculations is https://cran.r-project.org/web/packages/PropCIs/index.html. – whuber May 24 '22 at 13:41
  • @whuber sorry but its not really answering the questions, you're referring to the assumptions used for computing the absolute difference between two proportions. I'm referring to the difference in Percentage and adding a CI for that statistic which is completely different. – Roger Steinberg May 24 '22 at 13:51
  • It fully answers the technical issue of computing any of the CIs you have alluded to. Answers to the question about what a CI means and how to interpret it can be found in links already provided earlier in this comment thread. – whuber May 24 '22 at 15:16
  • @whuber but the technical issue presented in the question is "I'm trying to display the confidence interval of the % difference between p1-p2" correct me if im wrong but none of the proposed options above answer that. – Roger Steinberg May 24 '22 at 19:12
  • 1
    I still don't know for sure what you mean by "% difference," but for examples see https://stats.stackexchange.com/search?q=two+proportion+binomial+confidence+interval. – whuber May 24 '22 at 19:51
  • @whuber so confidence intervals as you suggest is clear(e.g. p1=0.2 , p2=0.18, diff = 0.02). That I can find the conf interval with binomial proportion confidence technique. However, what I want is use the % difference which would be -10% and I need the conf. interval for that -10%. – Roger Steinberg May 24 '22 at 22:12
  • 1
    Okay, so at a minimum please stop calling that ratio a "difference:" you're going to confuse most readers. Call it a ratio. The Normal approximations still work. Research delta method for details. – whuber May 24 '22 at 22:16

0 Answers0