4

I have two samples I'd like to do a hypothesis test on. The sample proportions are:

$$ \hat{p}_1 = \frac{120}{4000} $$

$$ \hat{p}_2 = \frac{9000}{300000} $$

If I did a test of two proportions where the null is $p_1=p_2$, would there be issues because $n_1=4000$ and $n_2 = 300000$?

Henry
  • 39,459
user321627
  • 4,474
  • 3
    One way to establish insight is to recognize that both numerators have approximately Poisson distributions and this result doesn't really depend on the sample size once the sample size is substantially larger than the numerator. Thus, when I need to conduct such a test quickly, I reason that "under the null, there are two independent Poisson variables with expectations near $\lambda n_x = 9000\lambda$ and $\lambda n_y=300,000\lambda$ where $\lambda = (120+9000)/(4000+300,000).$ In light of this, how likely are the observed counts of $120$ and $9000$? – whuber Dec 15 '22 at 22:27
  • @whuber Thanks, would this be an asymptotic test? If $X\sim Pois(\lambda n_X)$ and $Y\sim Pois(\lambda n_Y)$, would my p-value be $P(X=120, Y=9000)$? – user321627 Dec 15 '22 at 22:49
  • 1
    It would be an approximate test. That probability, which is guaranteed to approach $0$ as the sample sizes both increase no matter the outcome, definitely is not the p-value! See our thread on P-values for how a p-value would be computed by relating it to a critical region for the null hypothesis compared to the two-sided alternative. – whuber Dec 15 '22 at 23:26

2 Answers2

9

Similarity in sample size between groups is not an assumption for any hypothesis test for proportions that I'm familiar with. You should be fine to run the test.

Note: if you are planning a study, the study will be better powered (there will be a higher probability of finding a result if there is one) if the sample sizes in the two groups are similar. You don't want to plan a study and recruit 10 people in one group and 1000 in another group.

clementzach
  • 744
  • 2
    Equal sample sizes gives the best power if the costs of the two experimental conditions are equal. Often the treatment is much more expensive than the control, in which case the most power for a given total cost will result in an unbalanced experiment. Practical issues (capacity of treatment providers, manufacture of experimental medicines etc) may also limit the size of the treatment group. – JDL Dec 16 '22 at 09:30
4

There may be different issues here:

  1. very different sample sizes, here with one sample $75$ times the other: hypothesis tests for proportions will adjust for this automatically

  2. small proportions of successes, here of $0.03$: this can be an issue for small samples where different methods can sometimes give noticeably different results, but not really in this case with the large sample sizes

  3. large sample sizes, here in the thousands and hundreds of thousands: in general larger sample sizes are better, and the only risk is that you might get a statistically significant result which was very small and so not worth being concerned about - the answer to this is to consider a confidence interval for $p_1-p_2$ and in this example the 95% confidence interval for the difference would be approximately $[-0.005,+0.005]$

  4. invented numbers, as here the proportions are exactly equal and this is unlikely to be seen with such large samples: if the two population proportions were in fact both $0.03$ then the probability of the two sample proportions being equal would be about $0.0005$, which is very small, but not something to worry about here as you presumably invented these numbers simply to illustrate the question.

So none of these issues need worry you.

Henry
  • 39,459