5

So I've been learning about AB Testing and have used it to examine the form conversion rate of two different forms. However, I'm curious about testing whether a form with ads will generate more revenue than a site with just a form and no ads.

So the two things I'm comparing:

- form with ads
- form with no ads

I'm trying to see which produces more revenue.

So traditionally, a test of conversion rates would involve using a chi-square or fishers exact test to examine the values in a contingency table. I'm just not sure how to approach this question when it comes to revenue.

           Revenue     Conversions
Site 1 -    $500         100 
Site 2 -    $400         70

Is this no different that a tradition ab test of conversion rates?

Or could I just test wether a form made money or not.

          made money       made no money
form 1       50                200
form 2       5                 250
ATMathew
  • 2,645

2 Answers2

5

Since you're interested in revenue, it makes more sense to compare the revenue from the 2 sites rather than whether or not they made money. However, this means you are comparing distributions rather than contingency tables, and need different type of statistical test. One common test of this type is the t-test.

However, the t-test assumes that your data are normally distributed, which probably isn't the case in this situation. You can do a bit better by using a non-parametric test to compare your distributions, such as the Mann-Whitney U Test (also known as the two-sample Wilcoxon test).

In R, you can do this with the wilcox.test command.

Zach
  • 23,766
  • would it be possible to get an example in R of how you would set up a Mann-Whitney U Test with the asker's data? I am in the same situation but can't seem to get a useful result with wilcox.test. – bobfet1 Feb 14 '13 at 02:24
  • 1
    @bobfet1: I'm assuming the asker has 2 distributions of money, x and y. From ?wilcox.test you can see how to run this test. For example, if the null hypothesis is that the 2 distributions are the same, use wilcox.test(x, y, mu=0). If the p values is less than 0.05, you can probably reject the null hypothesis and conclude that the means of the 2 distributions are different. – Zach Feb 14 '13 at 17:31
3

To compare this two forms, you will need the number os users who filled this forms. More precisely, you need a sample from each form where the variables is the amount of money made with the user. Considering $\theta_{1}$ the vector of samples for the form with no ads and $\theta_{2}$, the samples for the form with ads, you can then do a hypothesis test on the sample mean: $$ H_0: \bar{\theta_1} = \bar{\theta_2}$$ $$H_1: \bar{\theta_1} \neq \bar{\theta_2}$$

Where the test statistic is: $$ z_0 = \dfrac{\theta_1 - \theta_2}{\sqrt{\dfrac{s_{\theta_1}^2}{n_1} + \dfrac{s_{\theta_2}^2}{n_2}}} $$ $n_1$ and $n_2$ are the sample sizes, $s_{\theta_1}^2$ and $s_{\theta_2}^2$ are the sample variance.

The null hypothesis is reject if $|z_0| > z_{\alpha/2}$, considering an $\alpha$ level of significance.

zeferino
  • 581
  • 2
    beware that you are assuming here that the data is (approximately) normal distributed ! Additional problem: If a visitor did not made a money, what is the value of this variable ? 0 ? Then the unknown distribution is not continuous anymore, too. If you on the other hand exclude those visitors, then the case of a form which creates less money but more frequently might be lost. – steffen Jan 17 '12 at 16:45
  • 1
    If the user did not made money, the value is 0. I am assuming that the distribution of the sample average of $\theta_1$ and $\theta_2$ are normal based on the CLT. – zeferino Jan 17 '12 at 17:08
  • What is "sufficiently large" given an unknown distribution with unknown parameters ? CLT should not be used as an excuse for not caring about the distribution ... but maybe it's just me. – steffen Jan 19 '12 at 14:38