0

I'm trying to teach myself bayesian a/b/c+ (more than 2 variations) testing and working my way through https://github.com/raffg/multi_armed_bandit/tree/master as an example.

My question is this: in a bayesian test of proportion with more than 2 variants, is there a pretty standard approach for evaluating the statistical significance of the difference between their posts? Can anyone point to a repo where I can see such stat sig eval for >2 variants in action?

I have seen the 2-variant eval used here: https://www.countbayesie.com/blog/2015/4/25/bayesian-ab-testing and I also understand how Google's potential value remaining approach would help you decide when to stop running the test. However, I'm not sure what else one should do, if anything, to monitor stat sig when there are more than 2 test branches to evaluate. Thanks for any advice.

ouonomos
  • 121

1 Answers1

1

My question is this: in a bayesian test of proportion with more than 2 variants, is there a pretty standard approach for evaluating the statistical significance of the difference between their posts? Can anyone point to a repo where I can see such stat sig eval for >2 variants in action?

There is no such thing as statistical significance for the Bayesian framework. Remember, significance is a single word to mean that a statistic computed under certain assumptions of the parameters is sufficiently extreme. Because Bayesians put probabilities on parameters, and not data given parameters, the concept of significance doesn't really apply.

Often, Bayesians will select an action which is expected to minimize some loss function. I demonstrate how to do this with a 3 variant AB test in this post here.

Demetri Pananos
  • 36,121
  • Thanks Demetri! So I hear you saying that the bayesian tester just evaluates the distance between the posteriors of multiple variants A/B/C... and combines this with a loss function. When the results are sufficiently far apart with small enough variance (circumstances permitting), and the loss is acceptably small, the tester judges the test complete. Is that correct? – ouonomos Oct 07 '23 at 23:20
  • Part of my confusion is that I see a p-value-ish process being used with Monte Carlo in examples like this https://www.countbayesie.com/blog/2015/4/25/bayesian-ab-testing#:~:text=Monte%20Carlo%20to%20the%20Rescue! . --And so I wonder if something analogous should be done for a bayesian test with more than 2 variations? – ouonomos Oct 07 '23 at 23:21