2

I'd appreciate help checking my basic understanding of how to use Bayes factor in a practical A/B/C test.

Is this general understanding of how to use Bayes' factors in A/B/C testing correct?

Let's say:

  • I am testing 3 versions of a web page: A= an unchanged control version, B= new version 1 and C = new version 2.
  • I create an identical prior distribution based on historical data, to be used for all 3 versions.
  • Now I run my test, yielding a posterior distribution for each version.
  • I decide that the test has run long enough. At this point, I can create 3 bayes factors, one for each version, by using the respective posteriors and the common prior.
  • Other things equal, I can then select the test version with the highest BF10 value (where 1 is the version and 0 is the original prior).
  • There is the somewhat strange possibility that the control would win, and be quite different to the prior, which I guess would mean both that the control click-through performs best and that I had a poor initial understanding of what the historical click-through rate was.

Is the above correct?

I understand there are other ways to pick a winner; I'd just like to understand whether I have the use of bayes factor for verstion test winner selection correct. Thanks for any advice!

ouonomos
  • 121

1 Answers1

0

I think my understanding above was bogus. bayes' factors are calculated from priors, and I had misunderstood bayes factors as being what I now know are posterior distribution ratios. So I would say to me above: "no, your understanding was wrong."

I would delete this question altogether, but there seems to be a penalty for doing so.

ouonomos
  • 121
  • Welcome to the site. It's actually a good question, no reason to delete it. There is plenty of opportunity for a better answer to come along, in part because I don't think "bayes' factors are calculated from priors" is fully correct or sufficient to explain it to future readers who might share your confusion. – shadowtalker Nov 08 '23 at 21:32
  • Thank you @shadowtalker ... having a little better handle on this topic now, I can see that there are a number of "why use bayes factors? why not use posterior distribution ratios?" threads in SE. So my initial question seems redundant; I'll keep it around though if people think it helps. Viz: https://stats.stackexchange.com/questions/547114/bayesian-hypothesis-testing-bayes-factor-vs-comparing-the-posterior-distributio. and https://stats.stackexchange.com/questions/229852/why-is-the-bayes-factor-sometimes-considered-more-important-than-the-posterior-o – ouonomos Nov 08 '23 at 22:59