2

Assume I aggregated the following results:

| Treatment | Count | Total | Proportion |
|-----------|-------|-------|------------|
| Control   | $c_n$ | $c_t$ | $p_c$      |
| Variant   | $v_n$ | $v_t$ | $p_v$      |

It seems like I can compute the $z$-score in one of two methods:

Option 1 / Pooled (link): $z$-statistic is given by $$z = \frac{p_c - p_v}{\sqrt{\frac{\hat{p}(1-\hat{p})}{c_t} + \frac{\hat{p}(1-\hat{p})}{v_t} }} $$ where $p_c$ is the proportion of the control group and $p_v$ is the one of the variant group. Furthermore, $$\hat{p} = \frac{c_t p_c + v_t p_v}{c_t + v_t}$$ is the pooled proportion where $c_t$ and $v_t$ are the sample sizes of the control and variant groups, respectively.

Option 2 / Unpooled (link): Here the $z$-score is given by:

$$z = \frac{p_c - p_v}{\sqrt{SE_c^2 + SE_v^2}}$$

where

$$ SE_c = \sqrt{\frac{p_c(1-p_c)}{c_t}} \quad ; \quad SE_v = \sqrt{\frac{p_v(1-p_v)}{v_t}} $$

My questions are:

  • Under what circumstances should I use the one or the other? My understanding tells me it depends on my knowing/assumption on the variance. If I assume it is the same or not big of a difference, then I should use the pooled and the unpooled otherwise.
  • When would the difference between the two be significant?
  • Is there an implemented test in Python where I can choose which approach to use? In statsmodels.stats.proportion it seems like only the pooled option is available.

Edit:

Dror Atariah
  • 279

1 Answers1

0

This answer was wrong (but can not be deleted because the OP has accepted it).

Sextus Empiricus
  • 77,915
  • What do you mean by "if you allow yourself to jump trough a lot of different hoops, and come up with an (more complex) alternative to the z-test"? Can you elaborate please? Is the choosing of pooled vs. non-pooled has to depend on the alternative hypothesis (one vs. two sided)? – Dror Atariah Feb 27 '18 at 12:23
  • The z that you calculate is only approximately standard normal distributed. The pooled version is a better approximate than the un-pooled version, so therefore it is better. But possibly you might use something else than the z-score and the normal distribution in the design of your tests (which will be complicated and that is the reference to the hoops), in which case the un-pooled version might provide more power (more likely to reject $H_0$ when $H_0$ is false). – Sextus Empiricus Feb 27 '18 at 13:55
  • Can you please add a pointer to your favorite reference discussing this point? – Dror Atariah Feb 28 '18 at 06:50
  • 1
    "The t-distribution is a (complicated) variation of the z-distribution." --> t-distribution is not a variation of z-distribution. It depends on whether sample or population variation is used. To test proportions, z-test instead of t-test should be used, as p and n are used to calculate/approximate the variance instead of independent resource of mean. – L.Yang Oct 04 '21 at 20:36
  • @L.Yang the t-distribution is a generalization of the Gaussian distribution in the following sense: It is a Gaussian distribution whose variance follows a chi-squared distribution. That is, if $X \sim N(0,\sigma^2)$ with $\sigma^2 \sim \chi_\nu^2$ then $X$ follows a t-distribution with $\nu$ degrees of freedom. – Sextus Empiricus Oct 04 '21 at 20:56
  • 1
    @L.Yang But re-reading my answer I see that I made a big mistake with adding the t-distribution into the story. – Sextus Empiricus Oct 04 '21 at 21:07