24

I am a software engineer looking to build an A/B testing tool. I don't have a solid stats background but have been doing quite a bit of reading over the last few days.

I am following the methodology described here and will summarize the relevant points, below.

The tool will allow designers and domain experts to configure a website to split traffic received at a specific URL between two or more URLs. For example, traffic arriving at http://example.com/hello1 could be split between http://example.com/hello1 and http://example.com/hello2. Traffic would be split evenly between target URLs and the performance of the marketing processes at each of the target URLs will be compared.

In this experiment, the sample size N will correspond to visitors. The test will measure "conversions", a term describing when a visitor commits to a specific action in a marketing process. Conversions are expressed in percentages and a higher conversion rate is desirable. This makes the test a comparison of independent proportions. The tool needs to be able to be easily employed to produce tests with safe results. Selecting an appropriate value of N is important.

In the linked article, above, a power analysis of two independent proportions is employed to find N. This method requires that one know the conversion rate of the control in advance as well as specify the target desired conversion improvement. It also specifies a significance level of 95% and a statistical power of 80%.

Questions:

  1. Is this method of determining N sound? If so, what is the safest way to determine the conversion rate of the control prior to beginning the test?
  2. Are there sound ways of determining N that don't require that one know conversion rates of the control in advance?
  3. Is the methodology in the linked article sound? If not, are there any accessible and easily digestible methods out there that you could link me to?
jkndrkn
  • 703

3 Answers3

13

The most common method for doing this kind of testing is with binomial proportion confidence intervals (see http://bit.ly/fa2K7B$^\dagger$)

You won't be able to ever know the "true" conversion rate of the two paths, but this will give you the ability to say something to the effect "With 99% confidence, A is more effective at converting than B".

For example: Lets assume that you have run 1000 trials down path A. Of these 1000 trials, 121 were successful conversions (conversion rate of 0.121) and we would like a 99% confidence interval around this 0.121 result. The z-score for 99% confidence intervals is 2.576 (you just look this up in a table), so according to the formula: $$ \begin{aligned} \hat p &\pm 2.576\left(\sqrt{\frac{0.121 \times (1 - 0.121)}{1000}}\right) \\ \hat p &\pm 0.027 \end{aligned} $$ So with 99% confidence we can say that $0.094 \le \hat p \le 0.148$, where $\hat p$ is the "true" conversion rate of process A.

If we construct a similar interval for process B, we can compare the intervals. If the intervals don't overlap, then we can say with 98% confidence that one is better than the other. (Remember, we're only 99% confident about each interval, so our overall confidence about the comparison is 0.99 * 0.99)

If the intervals do overlap, then we have to run more trials, or decide that they are too similar in performance to distinguish, which brings us the tricky part - determining $N$, the number of trials. I'm not familiar with other methods, but with this method, you aren't going to be able to determine $N$ up front unless you have an accurate estimate of the performance of both A and B up front. Otherwise, you are just going to have to run trials until you get samples so that the intervals separate.

Best of luck to you. (I'm rooting for process B, by the way).

$^\dagger$ The link doesn't work.

User1865345
  • 8,202
ronny
  • 154
  • 2
    Welcome to the site, @ronny. Since you're new here you may want to read our FAQ. Among other things, this site supports $\LaTeX$ via mathjax. I took the liberty of adding mj to make your post easier to read; make sure it still says what you want. EG, I took "p^" to mean "p-hat" ($\hat p$), but I note you say that it is the "true" rate, whereas p-hat is often used to indicate the p estimated from your data, so we just want to be sure your answer says what you want it to say. – gung - Reinstate Monica Oct 05 '12 at 17:01
  • ronny,you are right in general (not only for this procedure): the confidence intervals and consequently the sample size requirement are extremely sensitive to the true proportions for A and B. As a maybe more precise follow up to @gung's comment: I note that you (and consequently gung in his edit) use $\hat p$ for both the true proportion (in $0.094 \leq \hat p \leq 0.148$) and for the point estimate $\frac{sucesses}{trials}$ from the observation. I'd have written the upper two $\hat p$ (calculated from observation), but the lower two $p$ without hat (for the true proportion). – cbeleites unhappy with SX Oct 05 '12 at 18:50
  • 1
    This answer is incorrect. Specifically: "If the intervals don't overlap, then we can say with 98% confidence that one is better than the other" is wrong. Given two non-overlapping 99% confidence intervals, the confidence that the difference excludes 0 as at LEAST 99%. If the intervals are the same size, the difference is significant at around the 99.97% level. http://stats.stackexchange.com/questions/18215/ https://www.cscu.cornell.edu/news/statnews/Stnews73insert.pdf – Bscan Nov 03 '15 at 19:45
  • @Bscan Does your comment hold for other values? E.g. is it correct to say (according to your commend) that the difference of the means is at least 30% if we have two non-overlapping 30% confidence intervals of the same size? – Felipe Apr 19 '16 at 19:39
  • 1
    @Felipe, yes the comment holds for all values and non-overlapping 30% confidence intervals implies the confidence that the difference excludes 0 is at least 30%. This however doesn't mean there is a 30% difference in means. The true means may be very similar; we are simply trying to prove they are not exactly the same. – Bscan Apr 20 '16 at 17:45
9

IMHO, as far as it goes, the post goes into the right direction. However:

  • The proposed method implicitly makes two assumptions: the baseline conversion rate and the expected amount of change. The sample size depends very much on how good you meet these assumptions. I recommend that you calculate required sample sizes for several combinations of p1 and p2 that you think are realistic. That will give you a feeling about how reliable the sample size calculation actually is. 

    > power.prop.test (p1=0.1, p2 = 0.1*1.1, sig.level=0.05, power=0.8)

     Two-sample comparison of proportions power calculation 

              n = 14750.79
             p1 = 0.1
             p2 = 0.11
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

 NOTE: n is number in *each* group 

> power.prop.test (p1=0.09, p2 = 0.09*1.1, sig.level=0.05, power=0.8)

     Two-sample comparison of proportions power calculation 

              n = 16582.2
             p1 = 0.09
             p2 = 0.099
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

 NOTE: n is number in *each* group

    So if the actual conversion rate is 9% instead of 10%, you need another 2000 cases for each scenario to detect the 10%-more-than-baseline conversion rate of the new form.

After the test is done, you can calculate confidence intervals for the proportions based on your actual observations.

  • the last conclusion under 3. (about testing multiple scenarios) is not quite correct. To adjust for multiple testing (in the example multiple = 2), it is not enough to add just another $n$ tests for each new scenario:
    If neither B nor C are better than the original version A, and the two tests A ./. B and B ./. C are done as proposed there with $n$ cases for each of the scenarios, then the probability to falsely change away from A is (1 - α)² ≈ 10% (α: accepted probability of type I error; sig.level above). In other words, it is almost twice as large as specified initially. The second problem with that approach is: can you really do without comparing B ./. C? What are you going to do if you find both B and C better than A?
cbeleites unhappy with SX
  • 38,684
  • Hi, thanks so much for taking time to critique these methods. In the calculation (1 - α)² ≈ 10%, what does "α" refer to?

    Since capturing test data takes a long time, how do you propose one should construct this experiment if one wants to test three proportions? Is there a safe way to do so that doesn't involve running multiple tests? With three alternatives, three tests isn't terribly burdensome, but with four alternatives the number of combinations shoots up to six.

     – jkndrkn Oct 07 '12 at 15:41
  • 1
    @jkndrkn: α is the probability falsely to change away from the original form, aka α-error or type I error. See updated answer. – cbeleites unhappy with SX Oct 08 '12 at 12:23
  • 1
    @jkndrkn: Multiple tests: I'd have a look at Fleiss et.al.: Statistical Methods for Rates and Proportions about procedures for such tests. However, the key point for such multiple tests is always to use expert knowledge to cut down the number of alternatives as much as possible before defining the test because the required sample sizes explode with the number of alternatives (as you already realized). – cbeleites unhappy with SX Oct 08 '12 at 12:29
-1

Instead of calculating overlapping intervals you calculate the Z-score. This is algorithmically easier to implement, and you will get statistical libraries to help.

Take a look here.

User1865345
  • 8,202
Shambhu
  • 1