0

How to calculate sample size for online experiments when randomization and analysis unit are different?

For example, if we have an online learning platform for schools, and we make changes to make it easier for teachers to use, and we need to randomize at the school level so all teachers in the same school have the same software experience; however, we want to calculate the metrics at the teacher level (eg. pageview per teachers or % of teachers conducting specific actions). How can I conduct power analysis to determine the numberteachers we need and back into how many schools to include in randomization, assuming we have the alpha, power and MDE parameters set.

Usually I use the power analysis formula: $\frac{16 \;* \; \sigma^2}{\delta^2}$, but what should be the SD and delta be calculated with (teacher level or school level)?

user1456579
  • 31

1 Answers1

1
  • $\delta$ will be the analysis level, in your case is teacher level
  • $\sigma^2$ can be calculated by the delta method:

$$a(z) \sigma_X \sigma_Y /\mu_Y \approx a(\mu_X/\mu_Y) \sigma_X \sigma_Y /\mu_Y = \left( \frac{\mu_X^2\sigma_Y^2}{\mu_Y^4} - \frac{2 \mu_X \rho \sigma_X \sigma_Y}{\mu_Y^3} + \frac{\sigma_X^2}{\mu_Y^2} \right) ^{\frac{1}{2}}$$

more detail can refer to A/B testing ratio of sums

wei
  • 543