0

I recognize that the sample size calculation for the two-sample test at $\alpha = 0.05$ and $\beta = 0.80$ under a normal distribution becomes:

$$ n_{i} = \frac{16 * \sigma^2}{\Delta^2} $$

But, what is in variance in the context of proportion vs mean measures? If the experiment was an AB experiment, can I assume that the variance is derived from the variance of the current population?

I've also seen that $\sigma^2$ in a proportion test also becomes $\sigma_{1} + \sigma_{2} = p_{1}(1-p_{1}) + p_{2}(1-p_{2})$. But, when the test is a comparison of means, how do you know what's the appropriate variance?

Dan Lee
  • 11

1 Answers1

1

The variance estimate for comparing means between 2 groups is the variance within each group. (The standard 2-sample t-test assumes that both groups have the same within-group variance, although Welch's t-test relaxes that assumption.)

So you need to have a reliable estimate of that variance to estimate your sample size reliably. If you don't have such an estimate from your knowledge of the subject matter and the literature, a pilot study can help. Unfortunately, estimates of variance themselves have high variance, so you need to be conservative when working from a small pilot study.

EdM
  • 92,183
  • 10
  • 92
  • 267