0

I just started learning about A/B testing and I apologize if my questions may be weird. After reading/watching some content with regards to A/B testing, I cannot fully understand on how to evaluate (or the best/correct way) if your A/B test has a 'significant difference' result.

Let's say I have these results from my A/B test where I have 100 sample size for each group: A (control group): 20% conversion B (test group): 25% conversion

In this case, what statistical test should/can I use to evaluate this? Chi-squared test? Or is A/B testing supposed to be tracked daily over a period of time and use another statistical test to evaluate?

Appreciate if anyone can point me to a good example or the right direction in understanding this! thank you!

user3440156
  • 31
  • If you only have the percentages and not the raw numbers (e.g., if that $20%$ is $20$ of $100$ or $2$ out of $10$ or $2000$ out of $10000$), I do not see how you would be able to do a hypothesis test. Do you have those raw numbers? – Dave Jul 18 '21 at 04:12

1 Answers1

1

Conversion rates without sample sizes (i.e. counts of those who convert vs. those who do not) are not useful. 25% could mean 25 out of 100 or 2500 out of 10,000. The precision of the estimate in the latter is greater than that of the former.

If your design is a classic two groups-binary outcome, you can use any number of the tests I write about here. Significance here though is not really useful since large samples will nigh always yield a significant result. It might be better to analyze the expected loss vis a vis a Bayesian analysis.

Demetri Pananos
  • 36,121