0

When conducting AB tests, we use power analysis to calculate sample size with alpha, power and MDE (minimal detectable effect) parameters.

Mean MDE for continuous variable seems intuitive: Using Cohen's D to calculate the standardized mean difference = (M1-M2)/pooled SD

If I have proportions, what is a good way to calculate MDE? And how to work the other way around to know given we want to detect say 5% MDE, how to translate that to the actual relative change? Eg. Baseline conversion rate is 10%, we want to detect a relative 10% lift (aka 10% *1.1 = 11% conversion rate), what is the MDE? And if we want to detect 5% MDE, what is the relative change we can detect from baseline conversion rate?

user1456579
  • 31

1 Answers1

0

The minimum sample size can be approximated using the following formula

$ n = (\Phi^{-1}(1 - \frac{\alpha}{2}) + \Phi^{-1}(1 - \beta)) * \frac{\sigma^2}{\delta^2} $

In the commonly encountered scenario where $\alpha = 0.95$ and $\beta = 0.2$ (power = $1 - \beta = 0.8$). the formula simplifies to:

$ n = 16 * \frac{\sigma^2}{\delta^2} $

here, $\delta$ is the minimal detectable effect. It signifies the expected difference between the treatment group and the control group without normalization or standardization. In most cases, this value relies on your expectations or assumptions and does not require a calculation. So, in your first case, the minimal detectable effect is 0.11 - 0.10 = 0.01, and the second case is 0.005. $\sigma$ is the std err, if you are in a proportions test, then $\sigma = \sqrt{p * (1 - p)} = \sqrt{(0.1 * (1 - 0.1))} = 0.3$

wei
  • 543