Calculate confidence/chance of beating baseline

Question

As shown in the Google Web Optimizer example I would like to calculate the chance of beating baseline (original) - I tried around but couldn't figure out how they do it. I think the +- next to the conversion rate is the standard error but my standard error is off as well.

Anyone an idea how Google calculates it? Example in any programming language very welcome :)

Additional (but not important), if someone knows how they get the candlestick low close value that would be nice too.

Thank you!

Answer 1

Google Web Optimizer has been renamed Google Analytics content experiments, which use a testing strategy called multi-armed bandits.

Specifically, it offers Bayesian estimates of the best binomial bandit. Here's a paper written by a Googler:

• Steven L. Scott, A modern Bayesian look at the multi-armed bandit, Appl. Stochastic Models Bus. Ind. 2010; 26:639-658. (http://www.economics.uci.edu/~ivan/asmb.874.pdf)

Here's an R package based on the paper.