1

I have been searching a lot for this, in SO I have found some similar answers (1, 2) but they don't answer my question.

I'm trying to understand the math behind AB testing significance to answer some questions I have, but I don't have found any step-by-step computation. I know how to do a hypothesis test, the only problem I have with AB testing is that I don't have the standard deviation to do a normal hypothesis test (If you can do a hypothesis test without the standard deviation, I guess I actually don't know fully how to do a hypothesis test?). For example, if I run 3 Campaigns (C1,C2,C3) + 1 control campaign:

How could I know if one campaign is better than other at X% confidence level and what's the actual math behind the test?

I even found some calculators to compute the significance (SurveyMonkey, Investisdigital ), but none of them shows how they do it. So, does anyone knows how to do the actual computation?

The questions I hope to answer with the step-to-step computation (I hope this helps to know how specific/deep the computation might be):

  1. How can I know which of those three campaigns is better/worse with 95% confidence level than the control campaign?
  2. How the significance changes given a change in total?
  3. How is the test actually tested without the standard deviation?

EDIT:

I forgot to add that I also found a fisher test here applied to AB testing. However, I get lost when skips the process to the results from this matrix:

enter image description here

To its results:

enter image description here

Chris
  • 525
  • Because you're running multiple variants at the same time, and you want to know if one is "best", you're going to need to treat outcomes in each group as binomial and do a test of proportions. The math behind that is found here. Because of the multiple testing, you'd need to do a correction of the p value. See Bonferonni or similar. – Demetri Pananos Aug 18 '22 at 14:30

0 Answers0